DaDA: Distortion-aware Domain Adaptation for Unsupervised Semantic Segmentation
Authors: Sujin Jang
Joohan Na
Dokwan Oh
Venue: NeurIPS 2022 Accept (2022)
Keywords: unsupervised domain adaptation
relative distortion learning
semantic segmentation

Abstract: Distributional shifts in photometry and texture have been extensively studied for unsupervised domain adaptation, but their counterparts in optical distortion have been largely neglected. In this work, we tackle the task of unsupervised domain adaptation for semantic image segmentation where unknown optical distortion exists between source and target images. To this end, we propose a distortion-aware domain adaptation (DaDA) framework that boosts the unsupervised segmentation performance. We first present a relative distortion learning (RDL) approach that is capable of modeling domain shifts in fine-grained geometric deformation based on diffeomorphic transformation. Then, we demonstrate that applying additional global affine transformations to the diffeomorphically transformed source images can further improve the segmentation adaptation. Besides, we find that our distortion-aware adaptation method helps to enhance self-supervised learning by providing higher-quality initial models and pseudo labels. To evaluate, we propose new distortion adaptation benchmarks, where rectilinear source images and fisheye target images are used for unsupervised domain adaptation. Extensive experimental results highlight the effectiveness of our approach over state-of-the-art methods under unknown relative distortion across domains. Datasets and more information are available at https://sait-fdd.github.io/.

DaDA: Distortion-aware Domain Adaptation for Unsupervised Semantic Segmentation
Sujin Jang Samsung Advanced Institute of Technology s steve. jang@samsung. com
Joohan Na
Samsung Advanced Institute of Technology joohan na@samsung...... com
Dokwan Oh
Samsung Advanced Institute of Technology dokwan.. oh@samsung com
Abstract
Distributional shifts in photometry and texture have been extensively studied for unsupervised domain adaptation, but their counterparts in optical distortion have been largely neglected. In this work, we tackle the task of unsupervised domain adaptation for semantic image segmentation where unknown optical distortion ex- ists between source and target images. To this end, we propose a distortion-awe aware domain adaptation (DaDA) framework that boosts the unsupervised segmenta- tion performance. We first present a relative distortion learning (RDL) approach that is capable of modeling domain shifts in fine-grained geometric deformation based on diffeomorphic transformation. Then, we demonstrate that applying addi- tional global affine transformations to the diffeomorphically transformed source images can further improve the segmentation adaptation. Besides, we find that our distortion awwae adaptation method helps to enhance self-supervised learning by providing higher-quality initial models and pseudo labels. To evaluate, we propose distortion adaptation benchmarks, where rectilincar source images and fisheye target images are used for unsupervised domain adaptation. Extensive experimental results highlight the effectiveness of our approach under unknown relative distortion across domains. Datasets and more information are available at httpt::///// // //ssatttta fdd. fiiit-uudd.iituu..... github.ion
state-of- of-the-art methods
Introduction
Recent years have witnessed dramatic improvements in semantic segmentation performance based on a massive amount of pixel-level annotations. However, applying a segmentation model learned from the training dataset (source domain) directly to unseen test scenarios (target domain) often leads to significant performance degradation, which is caused by the divergence of data distribution between source and target domains. In addition, manually annotating target domain images with pixel-wise labels is extremely expensive and taxing. To address such issues, many studies have proposed unsupervised domain adaptation methods transfer the knowledge learned from the labeled
source domain to the unlabeled target domain (e.g., [33 34 27 26 (0)).
Existing works unsupervised segmentation adaptation methods have predominantly investigated domain shift problems mainly due to photometric and textural discrepancies between rectilinear image domains (e.g., adapting synthetically generated images [29 30 real-world imagery [8] or adapting images from/to different cities (7]). In contrast, domain shifts in geometric and optical distortion have not been well explored, despite commonly appearing in many practical applications. For example, wide-angle cameras (e.g.. fisheye cameras) have been extensively used for complex
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Trem
Resampling
Frnt
Source Us)
Sealing Squaring
Source (2g
Transformed
(
Person
Person
M(1
Tagget
Transformed
M(
Car
Teaget (1y .g)
W/o DaDA
w/ DaDA (ours)
Distortion-aware domain adaptation (DaDA) framework.
(b) Class-activation visualizations
3T
Figure 1: Overview of DaDA framework and qualitative results. (a) We depict the relative distor- tion learning (blue) and the segmentation adaptation (red) with related loss functions. Deformation field generator G produces a flow field u(()0) to transform source data (Tg) to new image (Js-r) replicating the distortion style of the target image (Tr). (b) Our framework shows stronger and finer boundary of class-wise activation visualizations under unknown radial distortion across domains (more illustrations are shown in Appendix C).
vision systems like autonomous vehicles 38 and surveillance 4 to obtain more information about the surrounding environment. Fisheye images (e.g., target image IT in Fig have quite different geometric deformations (e.g., radial distortion) compared to regular rectilinear images (e.g., source image 1g in Fig D. Such distortion variants pose even more challenging domain adaptation tasks. For example, a state-of-thee thete-r-tt semantic segmentation model 6 only trained on rectilinear images fails to correctly predict pixels of vehicles and roads under optical distortion and its overall prediction accuracy drops from 67.02% (trained on fisheye images) to 32.39% (trained only on rectilinear images, see T. To alleviate this, we may rectify distortion test time. However, such a workaround inevitably leads to reduced field-of-view (e.g., over 30% loss for fisheye images (38) ) resampling distortion artifacts the image periphery, calibration errors in practice, and additional computing resources for the rectification step at test time [19], which is against the original purpose of using wide-angle cameras. This urges us to use native fisheye images without needing rectification. Another motivation for our work is the scarcity and difficulty of constructing annotations for distorted fisheye images (e.g., Woodscape [38]. while we already have larger amounts of annotations for rectilinear images (e.g., Cityscapes R. GTAV 291). Remarkably, we have relatively fewer public datasets providing large-scale and finely annotated fisheye images. Woodscape (38] is currently the only real-world public dataset with segmentation annotations. Such lack of annotations for distorted images necessitates introducing optical distortion into unsupervised domain adaptation.
With these insights, we first formulate an important but challenging unsupervised domain adaptation task for semantic segmentation where unknown optical distortion exists between the source and target domains. To this end, we propose a novel distortion-aware domain adapiation (DaDA) framework, which provides new perspective to minimizing domain gaps in geometric and optical distortion. We first present relative distortion learning (RDL), which is capable of modeling relative deformation between the source and target domain. In particular, we build deformation field generator to transform the source image new image sharing similar distortion features of the target image. To enable such challenging unsupervised and unpaired distortion learning, we exploit the properties of diffeomorphism, that is differentiable and has a differentiable inverse. We directly integrate such properties into our distortion-awad -aware losses to enforce the semantic quality of relative deformation fields at the image- and the prediction level. We also observe that applying additional global affine transformations (e.g., rotation, shearing) to the diffcomorphically transformed source images can further improve the segmentation adaptation performance in most cases. Owing to fine-grained diffcomorphic and global affine transformations, our framework provides distortion- aware segmentation adaptation models and reliable pscudo labels, and thus ultimately improves the performance of self-super learning methods.
To validate, we propose new domain adaptation benchmarks, where segmentation model trained on rectilinear images (Cityscapes (8 or GTAV (29 is transferred to fisheye images (Woodscape [X or our in-house fisheye driving scene dataset (FDD)). With mean Intersection-over-UUu (mloU) as the evaluation metric, our framework achieves significantly improved prediction performance compared to existing segmentation adaptation methods.
2

---

In essence, our key contributions can be summarized as: (1) a distortion aware domain adapta- tion framework for boosting unsupervised semantic segmentation performance in the presence of unknown relative distortion; (2) distortion-aware losses that effectively enforce the unpaired and unsupervised relative distortion learning to transfer distortion style from target to source domain; (3) new unsupervised domain adaptation benchmarks posing challenging tasks, where source and target images have additional domain gaps in optical distortion.
2
Related Work
2.1 Domain Adaptive Semantic Segmentation
Recent advances in image-to-iman translation and style transfer 42 have promoted the translation of source-domain images to mimic texture and appearance of the target-domain images. CyCADA (24] applied CycleGAN [42] to preserve semantic consistency before and after image-level adaptation. BDL [22) proposed bidirectional learning method where image-to-ine -mage translation and segmen- tation adaptation is learned alternatively. Similar methods have been proposed to adapt source images 35 17 reconstruct the source image from the output-level representation [36]. Deviated from these adversarial approaches, FDA (37 proposed a method to align the low-level frequencies of source images and their counterparts in target images. Ma al. 25 applied photometric alignment method to minimize domain shifts in image- and category-level feature distributions.
More recently, self-supervised learning (SSL) approach has emerged in domain adaption tasks to further improve the segmentation adaptation performance. SSL approaches try to rectify noisy pseudo
labels with respect to uncertainties confidence scores [22] 17 28 27 26 13
40 These
methods inherently require good initial models, that are commonly trained on source or adapted on target beforehand. To obviate the need of multi-stage and adversarial training, Araslanov and Roth 2 introduced a photometric augmentation method to SSL.
However, none of the prior works directly considers distortion oriented domain shifts in the unsuper- vised segmentation adaptation tasks. Moreover, it remains unknown how well existing adaptation methods address unknown relative distortion between domains. As pioneering work, we first aim to evaluate prior arts on the adaption task the presence of unknown optical distortion across domains.
For this, we noted that many of current state-of-the-art methods rely on SSL (e.g., [17 (7] 26] 41 400)
and have adopted AdaptSeg [33 or AdvEnt (34) as for their based segmentation adaptation. Thus, baseline methods and further evaluate SSL approaches [26] 27 40 in the presence of optical distortion shifts across domains. To evaluate, we newly formulate segmentation adaptation benchmarks, i.e., transferring from real synthetic rectilinear images to real fisheye images.
take these ad versarial adaptation approaches
2.2 Diffeomorphic Deformation Networks
Spatial transformation networks (STN [15) have been used in various contexts since it is capable of learning differentiable deformation field. STN primarily allows simple linear transformations (e.g. affine, translations, scaling, and rotations) and can be extended more flexible mappings such as thin plate spline transformations [23). However, optical distortion involves complex nonlinear transformation and STNs limited support such transformations. Instead Detlefsen et al. [12 proposed to exploit diffeomorphisms in spatial transformation, which can model complex nonlinear deformation and address optimization divergence issues in learning spatial transformation. A dif- feomorphism is a globally one-to-one continuous and smooth mapping, which has a differentiable inverse. Diffeomorphic transformations have been largely used for image registration [16] 10 32) and shape analysis (39). However, traditional diffeomorphic deformation methods demand high computational costs and difficult implement; and thus make it difficult to be incorporated into deep neural networks. To address such issue, Dalca et al. [1]] proposed probabilistic generative model to generate diffeomorphic deformation field for medical image registration task. However, their work depends on a paired set of 3D brain images depicting the same contexts. Inspired by [1I, we propose a new relative distortion generator, which takes set of unpaired source- and target-domain image, to transform the source image to new image sharing similar distortion style of the target image.
3

---

2.3 Radial Distortion Rectification
Traditional distortion rectification approaches require images captured with specific camera calibration parameters [2)] and thus are not flexible. The other work exploits the principle that a straight line should be projected into straight line in a calibrated image [. but such method mainly depends on the accuracy of line detection. More recently, deep neural networks have been used for robust and efficient rectification performance 21). However, these works primarily aim to rectify distorted images and do not support distortion style transfer for unpaired and unsupervised domain adaptation tasks, In contrast, we propose to transfer distortion style between unpaired source and target images.
3 Method
Let S be the source-domain data with segmentation labels Ys and T be the target-domain data with no labels. We aim to train a scene segmentation network M performing satisfactory pixel-wise prediction on T by minimizing domain gaps between S and T. Here we formulate more challenging adaptation tasks where distributional shifts include not only visual domain gaps (e.g., texture, lighting, contrast) but also geometric and optical distortions (e,g., radial distortion). Such domain gaps make M difficult to learn transferable knowledge in both visual and geometric domains.
3.1 Relative Distortion Learning
To minimize the discrepancy between source and target domain, prior works proposed pixcl-level
image translation methods based on yyle-cconss [14 17 35 42 or photometric alignment [2] 251.
However, these methods do not consider domain gaps in geometric distortion and thus fail to transfer distortion style from Ir to Is (see Fig To address this, we propose relative distortion learning (RDL) method, which predicts a relative deformation field to transfer the distortion style between domains. Given a source-domain image Is and a target-domain image Ir, we aim to transform Is to new image Is-T based on a relative deformation field $5-TT where Is- shares a similar distortion style of IT. We achieve this transformation through grid-based sampling operation I5 HS-T Is- Ultimately, the transformed source image Ts--7 aim to mitigate the domain shift in optical distortion at a fine-grained level. However, it is not trivial to predict relative deformation fields since input images are not paired (i.e., dissimilar image contents); and the relative distortion involves nonlinear geometric deformation, which is hard to be parameterized without knowing optical features (e.g., lens distortion model, focal length). To learn such challenging geometric relationship, we exploit diffeomorphic transformation in the unpaired and unsupervised distortion translation task.
Diffeomorphic Transformation. Let u E 22xxxx be flow field which is constant over time. Then, we describe a differential equation of evolution of deformation (5()) by
th = u()
(1)
at
where t is time and (0)) the identity transformation. By integrating u()l over E [0, 1], we get a diffcomorphic deformation field $- == (11), which maps the coordinates from one image to another image. For fast and differentiable integration, we use exponentiated integration technique, so-called scaling- serhling-nnd-saae -ad--anddd--a and-squaring B. This integration method starts from ((1/2") ((0) + u/2"
and iteratively computes deformation fields of next time steps via (1/22 1) = 6(122)) (1/22) over
T' times, where indicates a grid-based resampling operation. Then, we obtain an approximate deformation field Φ. The inverse deformation field Φ- can be achieved by integrating the negative field (11/2") (0) u/2" via the same integration method.
To generate field of Ps-r,, which defines relative distortion between Is and ly, we propose deformation field generator G, which takes both Is and Iy: and their first-order gradient (i.e., Sobel filter) Vls and Vlr as input. Here we use the image gradients to provide rich geometric features inspired by the distortion rectification method [I. Then, G generates deformation field $s and its inverse field &T- For example, in Fig 2 generates a flow field u to construct a & (red-colored grid in (e)) from pair of IT (a) and Is (b). Finally, new transformed image Is--T (e) is generated via Is o $S-TT. Similarly, negative integration of the flow field u yields an inverse deformation field PT which generate reconstructed image (d) via I's Is-T @T

---

To supervise the generator G for learning the relative distortion in unpaired and unsupervised settings, we exploit the properties of diffeomorphic transformations: topology preserving, invertible and differentiable. Such diffeomorphic constraints are directly built into relative distortion learning to enforce desirable deformation field outputs. In particular, we propose distortion-awan losses: distortion reconstruction loss and semantic distortion-consiiter loss. These losses evaluate the cycle- consistency of the relative deformation field at the image- and the prediction-level, while enforcing the semantic quality of the relative deformation field. Distortion reconstruction loss (Crecon) ensures that generates convincing flow fields to reconstruct the source image Is from the transformed image IT by the inverse deformation field Ør-s, and vice versa:
Crecon = 11 1'sll1 + 1 - 11, where I's = 1s-T TT-S, 1 15
(2)
Semantic distortion-conee consistent loss (eeem) enforces semantic consistency between the source image Is and its transformed pair Is-r the pixel-wise prediction level. The same constraint can be applied to between IT and IT That is, an ideal deformation field generator G should be able to generate &ss-T and & that reconstruct structural information from the pixel-wise prediction outputs of the distorted input images. With these semantic constraints, the relative distortion learning can be further improved by the segmentation model M; and also the segmentation adaptation can benefit from the improved quality of relative deformation fields. The loss function can be written as:
Seem (Is) o p--T - M(Is-T)llI + (IT) o - M(IT-s)ll..
To minimize domain gaps in optical distortion between the transformed source images Is-T and the target-domain images Ir, we introduce a distortion-aware discriminator DG, which aims to discriminate distortion style between IT and Is-r7. Similar to G, we again the first-order image gradient, V1 and VIs+rr as input to DG along with the images. The loss function is defined as:
De E1g~8 1r Da $s-T, V o &s-T))) (I,-T(Da(Ir, Vlr)). (4)
We also calculate the adversarial loss using Do:
Lado = -E5--S, Da S- V &s-r)))l
(5)
where the deformation generator G tries to produce the relative deformation field to fool the discrimi- nator DG in distinguishing distortion style.
Therefore, the total loss function for the relative distortion learning is defined as:
Cra B₁ Crecon + 32m + ByLodv_G.
(6)
where 31, 82, and Bg are constants controlling the effect of corresponding losses.
3.2 Distortion-awarc Adversarial Adaptation
Relative distortion learning aims to adapt optical distortion shifts both pixel- and feature-level. Based on the adapted representations, baseline adaptation methods, we take AdaptSeg [33] and AdvEnt [34], that have been extensively used in many prior work 22 40) as for their based segmentation adaptation. Typically, the seg- mentation loss uses the cross-entropy for the transformed source image Is-IT and its corresponding label Ys-r!
further apply adversarial segmentation adaptation. As
Coeg - E
(h.w.c) S-T log(Al (Is-T)(()w.)))
h,W
where Ys-T is obtained by transforming ground-truth annotations from the source domain by Ys 45 Entropy minimization loss further introduced by AdvEnt (34]:
Lent
-1
ΣΣ h, GEC
M(IT) (h.)..) log M (Ir (Ir) (hran.))
Tog(C')
(8)
which tries to directly minimize pixel-wise entropies to enhance prediction certainty in the target domain. As a common practice, a domain discriminator Dar is used to minimize the difference between the transformed source and target prediction probabilities:
LDM = RBgg-S, - DM (M(Ts o &s-Tr)) + E,,
(9)
5

---

The adversarial loss function for segmentation adaptation can be written as:
Lade_M M D (IT))),
(10)
where the segmentation model M is trained to fool the discriminator DM-
Equipped with the relative distortion learning and the adversarial adaptation, we train the adaptive segmentation network with the following total loss:
Lall
Lydt + Ladv M Leeg + YLents
(11)
where y is 0 for AdaptSeg
33 and for AdvEnt 34
4 Experimental Results
We present extensive experimental results to validate our distortion-aware aware domain adaptation (DaDA) framework for semantic segmentation in the presence of both visual and geometric domain shifts. For this, we formulate new domain adaptive segmentation benchmarks, iee,, transferring from real-world synthetic rectilinear images real fisheye images.
4.1 Datasets
In the experiments, we first show evaluations of the model trained on real-world rectilinear images from the Cityscapes [8] and test the adapted model real fisheye images from Woodscape (38) or our in-house fisheye driving dataset (FDD). Then, we introduce a more challenging adaptation task, where the model trained synthetic dataset GTAV [29] is transferred to real fisheye images (Woodscape or FDD) without annotations. The Cityscapes dataset contains 2, 975 training images of high-quality driving scene with the resolution of 2048 1024. The GTAV dataset contains 24, 966 synthesized images with the resolution 1914 x 1052. The Woodscape dataset consists 8, 234 fisheye images with the resolution of 1280 966, where the images are captured by fisheye cameras (1909 F.O.V) looking at four different directions of the vehicle. We use front and rear camera scenes containing 4, 023 images in our experiments. The images are randomly split into a training set with 3, .22 images and validation set with 1, 000 images. We report the results on 17 classes aligning mismatched classes between Cityscapes and GTAV; and Woodscape. Our in-house fisheye driving dataset (FDD) includes 3, 897 of fully annotated images with the resolution of 1920 x 1080 captured by fisheye cameras (200 F.O. V) at front- and rear-side of the vehicle. We randomly pulled 974 of validation images and remaining 2, 923 images are used for the training. For FDD, we use 12 classes, where incompatible classes in Cityscapes and GTAV are merged or excluded similar Woodscape (see Appendix B. for detailed class information of Woodscape and FDD).
4.2 Experimental Details
Diffeomorphic and Affine Transformation
Fig depicts an example of diffeomorphic and affine transformations applied to source- and target-domain image. First, both original target and source images randomly cropped and re-
Randomly Cropped Images
sized to (a) and (b) followed by randomized hor-
izontal flipping and photometric jittering. For
the fine-grained diffeomorphic transformation, both target (Ir in (a)) and source image (1s in (b)) are used to generate transformed image Is-r (e) via the relative deformation field generator G. Additional global affine transfor- mation is applied Is-rr to generate an image in (f), which includes both fine-graines diffeo- morphic and global affine deformations. For the affine transformation, we adopted RandAug- ment (RA) [9] excluding preceded photometric jittering. RA is one of the immediately probable
(a) Target Image fr
(b) Irage
(e) +
Transform
(d) Reconstructed Source I's
(e) Transformed Source Iç-r
(0 is-r Affine Transform
Figure
An example diffeomorphic and affine transformations of training images.

---

Table 1: Comparisons with the baseline adaptation methods. All the methods are based on DeepLab- V2 with ResNet- 101 as the backbone for a fair comparison. Oracle performance (trained on target) Woodscape is 67. .02% and for FDD is 61.59%.
Cityscapes
GTAV
Cityscapes
GTAV
Woodscape
Woodscape
FDD
FDD
Method
mloU(%)
gain
mloU(%) gain
mloU(%)
gain
mloU(%)
gain
SourceOnly
32.39
29.32
34,76
32,13
AdaptSeg 33
46.33
35.94
39.07
36,90
Ad
50.44
+4.11
36.88
+0.94
39,42
+0.35
37.22
+0.32
AdaptScg+RDL
50.88
+4.55
37.36
+1.42
41.35
+2.28
39.29
+2.39
AdaptSeg+RA+RDL
52.59
+6.26
37.73
+1.78
41.07
+2.00
39.64
+2.74
AdvEnt R5
45.26
34.70
38.87
37.25
AdvEnt+RA
50.60
+5.34
36.64
+1.94
41.58
+2.71
38.75
+1.50
AdvEnt+RDL.
50.94
+5.68
36.39
+1.69
42,43
+3.56
39,93
+2.68
AdvEnt+RA+RDD
52.64
+7.38
37.62
+2.92
42.32
+3.45
40,87
+3.62
Cityscapes > Woodscape
GTAV > Woodscape
Ground-Truth
AdaptSeg Only Cityscapes > FDD
AdaptSeg
DaDA
Ground-Truth
AdvEnt Only GTAV > FDD
AdvEnt DaDA
Ground-Truit AdaptSeg Only
AdaptSeg DaDA
Ground-Truit
AdvEnt Only
AdvEnt DaDA
Figure 3: Qualitative examples. DaDA rectifies erroneous predictions, especially in the image periphery where severe optical distortions appear compared to the based adaptation methods.
and applicable augmentation methods including series of affine transformations (i.e., rotation, shear-x, shear-y, trans-x. and trans-y with 0.5 probability of application). For example, rotation transformation applied the source image Is (c). our experiments, we tested transformed images generated by either one of affine-only (c), diffeomorphic-on/ (c), or both transformations (f).
Implementation Details. We trained all networks with the Adam [88 solver with batch size of 4. The learning rate is 0.2 x 10-5 for M and DM and 0.1 x 10-6 for and De- We set the weight factors of losses in 66 6 B1 100.0, B2 10.0, B3 10.0 for Cityscapes
Woodscape (or FDD); and 3 100.0, 32 1.0, B3 = 100.0 for GTAV + Woodscape (or FDD). Further details on the implementation well our hyperparameter selections are provided in Appendix A and B.
4.3 Comparisons with State-of-the-Art Methods
We first evaluate the effect of our distortion-awarr adaptation framework when applied the based adaptation methods: AdaptSeg 33 and AdvEnt (34). We use DeepLab- V2 (6) with ResNet- 101 backbone as the base semantic segmentation architecture M. We test the different spatial transformations (+RDI: applying diffeomorphic transformation learned by RDL, + +RA: applying an affine transformation via RandAugment [9. +RA+RDL: applying both diffeomorphic and affine transformations) on source images in the adaptation tasks.
Woodscape as target domain (Tab(I). Compared the based adaptation methods, applying either one of ine-igri diffeomorphic (+RDL) or global affine transformations (+RA) achieves clear improvements on the all adaptation tasks. Remarkably, applying both fine- ane-grained diffeomorphic and global affine transformations (+RA+RDLL achieves significant improvements of +7.38% and +2.92% on Cityscapes-A - and GTAV - Woodscape tasks, respectively. Note that the performance improvement of the distortion-avee aware adaptation comes from object classes such as person (+20.07%) car (414.49%), bus (+1 (+114.1%% %),, and truck (+ 14.66.1% ()) as well as background such sidewalk (+13.09%) (see class-wise iou (%) in Appendix B.4). Optical distortion gradually

---

Table 2: Effect of DaDA on Self-Supervised Learning (SSL).
Cityscapes Woodscape
GTAV
Cityscapes
GTAV
Woodscape
- FDD
FDD
Method
+DaDA
mloU(%)
gain
mloU(%)
gain
mloU(%%
gain
mloU(%)
gain
47.00
38.83
39.60
37.47
IAST 26
53.82
+6.82
40.75
+1.92
44.46
+4.86
40.06
+2.59
48.92
36.10
40.36
38.61
IntraDA 27
53.24
+4.32
39.85
+3.75
45.28
+4.92
42.10
+3.49
50.69
34.44
39.72
35.97
ProDA 40
54.83
+4.14
35.75
+1.31
42.14
+2.42
37.09
+1.12
Target Image (lz) Source Image (s)) CycleGAN [44]
RDL (Is-r)
P--55
Is-r Tr-s
Figure 4: Comparisons relative distortion learning (RDL) with an image-to-image transla- tion method [2] (Top row: GTAV >> Woodscape, Bottom row: Cityscapes Woodscape).
increases when objects and back appear closer to the image periphery. We hypothesize that our distortion adaptation method rectifics erroneous predictions in such distorted regions. These improvements also confirmed in Fig (b). Fig and Fig (Sec Section 4.4))
FDD target domain (Tab Here, the results are consistent with the previous adaptation tasks. Again, +RA+RDL clearly shows superior prediction performance to the based methods by up to +3.45% and +3.62% on CFtD and -TAV~+FDD respectively. Notably, + +RDL contributes up to 3.56% (AdvEnt+RDL on Cityscapes - FDD) of improvements, which also consistently achieves higher performance gain than +RA (+2.71%). Such results are also echoed in the Cityscapes + CitysapessFishEEe task in Appendix B.2. Note that +RDL always leads to improvements in segmentation adaptation, regardless of domain shift, throughout our experiments (e.g., based methods +RDL, and +RA +RA+RDL) presented in Tab and Tab.7 in Appendix B.2. In contrast, the randomized affine augmentation (RA) leads to degraded segmentation adaptation results upon the geometric distributional shifts between source and target domains (e.g., +RDL vs. + RA in Cityscapes -> CitsscapesFisheye and Cityscapes - FDD). Thus, we may conclude that learnable diffeomorphic transformation (RDL) plays an important role in aligning the domain gap of geometric deformation.
Relationship to SSL (Tab()). We evaluate the effect of our distortion-awarr domain adaptation (DaDA). as a "warm-up" phase, for state-of-the-are adaptation methods using self-supervised learning: IAST 26 IntraDA (27), and ProDA [40] In particular, we train the initial segmentation model with + RDL distortion adaptation; and use AdaptSeg-baat based adaptation models for IAST (26) and ProDA [40] and AdvEnt-basee models for IntraDA (27]. Tab shows the effectiveness of DaDA, where it further improves the self-supervised learning by providing higher-quality initial models and pseudo labels. DaDA attains up to +6.82% improvement against the baseline SSL methods. This implies that satisfactory distortion-awant adaptation cannot be achieved only by relying on SSL.
Comparisons with Image-to-Image Translation. We compare our relative distortion learning (RDL) with an existing image-to-image translation method CycleGAN (42. Unpaired rectilinear source images (Cityscapes and GTAV) and fisheye target images (Woodscape) are given to both RDL and CycleGAN. The input images are randomly cropped and resized to 512 x 512. Fig shows that CycleGAN fails to generate transformed images from source images mimicking the distortion style of

---

Table 3: Ablation results on the distortion-aware losses.
Base Method
+Lade G
+Lsem
+recon
Cityscapes - Woodscape GTAV Woodscape
46.33
35,94
49.61
36.45
AdaptSeg
/
50.29
36.75
49.97
37.17
50.88
37.36
45.26
34.70
47.77
35.36
AdvEnt
34
49.22
35.77
50.32
36.11
50.94
36,39
target images. This is obvious to observe since CycleGAN-like existing translation approaches do not have devices (e.g.. diffeomorphic transformer) to geometrically transform source images. In contrast, RDL enables modeling relative distortion across domains and generates transformed source images alike the distortion style of target images. In Fig 4 buildings and vehicles distorted replicating counterparts in target images via RDL., while CycleGAN focuses on translating texture and color of images. Fig 4 also demonstrates that RDL is able to reconstruct source images (Is) from the distortion-tranns translated images (Is+T) via an inverse relative deformation field r Overall, these results imply that our distortion-awas losses (Crecons Ssmm) are effective in guiding the generator G to produce convincing relative deformation fields.
4.4 Ablation Studies
To better understand the effect of our distortion- adaptation approach, we conduct ablation studies on the distortion-aware losses and the competence in predicting distorted image regions.
Effect of Distortion-aware Losses (Tab3). We evaluate the effect the distortion-awarn losses on segmentation performance. deb[ids the improvement of prediction performance compared to the based methods, by adding individual or all of the proposed distortion-aware losses in Eq. [1). Only adding the adversarial loss (+Ladv G) contributes the adaptation performance up +3.28% of improvements. Progressive introduction of the distortion-aware losses consistently improves prediction accuracy (+Csem, +Crecon). Ultimately, utilizing all losses together with segmentation adaptation achieves +5.68% and +1.69% of improvements for Cityscapes + Woodscape and GTAV Woodscape, respectively. Here we observe relatively smaller improvement in GTAV + Woodscape task, which exhibits both severe visual and geometric distributional misalignment. We believe that introducing additional rexture-aware translation methods along with our distortion adaptation approach might lead to further improvement in such synthetic-to- to-rell adaptation task. Overall, the relative distortion learning supervised by the distortion-awe aware losses effectively reduces domain shifts in optical distortion, and thus improves the prediction performance.
Distortion-aware mloU. To quantita- tively demonstrate the effectiveness of DaDA on predicting distorted image areas, we propose a distortion-aware mloU(%) metric. The image coordi- nates are normalized to [-1.0,1.0] and we gradually mask label where /12 + of pixel at (i, j) smaller than certain dis- tance threshold (dāst). In Fig 5 (bottom row), dist 0.0 shows that the original label is used for mloU(%) calculation, while dist 0.8 indicates that large ar- eas of undistorted regions are masked out the label so that we evaluate the mod- els on distorted regions. Plots in (to row) show that the performance gain (AmloU) achieved by adding DaDA increases as dist increases. This indicates that DaDA effectively addresses domain shifts in distortion and improves the prediction performance for the distorted image regions.
Cityscapes Wondscape
Woudscupe
Adaptteg+DaDA AdDL
12.5
AdvEat-DaAA
11.0
5.1
8.8
3.0
0.2
0.3
0.4
055
0.6
0.7
6.8
0.2
0,3
0.4
0.5
0.6
0.7
0.8
Distance Threshold (diel)
Distance Threshold (diet)
dis
dist-0.2
disr-0.4
dis--0.6
dist
Figure 5: Distortion-aware mloU.

---

Discussion and Conclusion
In this paper, we proposed a novel distortion-aweer aware domain adaptation (DaDA) framework that is capable of modeling domain shifts in geometric deformation based relative distortion learning (RDL.) method. Besides, we demonstrated that our distortion adaptation approach further improves self-supervised learning by providing higher-quality initial models and pseudo labels. Extensive experimental results proved that our method minimizes domain shifts in optical distortion, and thus significantly improves the segmentation adaptation performance under unknown relative distortion across domains. In the future, we will investigate the interplay between the texture-oriented and the distortion oriiente domain shifts further improve the unsupervised domain adaptation. While we first tackle adapting existing semantic segmentation models trained on rectilinear images to unlabeled fisheye images, various set-ups of domain adaptation tasks among distorted and rectilinear images can be further considered. For example, we may use distorted images source and rectilinear images as target, or both source and target domains include distorted images. Applying relative distortion learning to such extended adaptation tasks could be another interesting direction for future work. We hope our work provides a solid baseline and new perspectives on distortion-awan aware domain adaptation.
6 Disclosure of Funding
We received no third-party funding for this work.
References
[1] Alemán-Flores, M., L. Alvarez, L. Gomez, and D. Santana-Cedrés (2014). Automatic lens distortion correction using one-parameter division models. Image Processing On Line 4, 327-343.
[2] Araslanov, N. and S. Roth (2021, June). Self-supervised augmentation consistency for adapting semantic segmentation. In Proceedings of the JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15384-15394.
[3] Ashburner, J. (2007). A fast diffeomorphic image registration algorithm. Neuroimage 38(1), 95-113.
[4] Boult, E., X. Gao, R. Micheals, and M. Eckmann (2004). Omni-directional visual surveillance. Image and Vision Computing 22(7), 515-534.
[5] Chao, C.-H., P.-.. Hsu, H.-Y. Lee, and Y.-C. F. Wang (2020). Self-supervised deep learning for fisheye image rectification. In Proceedings of IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 2248-2252.
[6] Chen, L.-C., G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille (2017). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence 40(4), 834-848.
[7] Chen, Y.-H., W.-Y. Chen, Y.-T. Chen, B.-C. Tsai, Y.-C. Frank Wang, and M. Sun (2017). No more discrimination: Cross city adaptation of road scene segmenters. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 1992-2001.
[8] Cordts, M., M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele (2016). The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 9223-3223.
[9] Cubuk, E. D., B. Zoph, J. Shlens, and Q. V. Le (2020). Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IERE/CVF Conference Computer Vision and Pattern Recognition Workshops, pp. 702-703.
[10] Dalca, A., M. Rakic, Guttag, and M. Sabuncu (2019). Learning conditional deformable templates with convolutional networks. Advances in neural information processing systems 32.
10

---

[11] Dalca, A. V., G. Balakrishnan, J. Guttag, and M. R. Sabuncu (2018). Unsupervised learning for fast probabilisticc diffeomorphic registration. In Proceedings of International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 729-738. Springer.
[12] Detlefsen, N. S., O. Freifeld, and S. Hauberg (2018). Deep diffeomorphic transformer net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4403-4412.
[13] Guo, X., C. Yang, B. Li, and Y. Yuan (2021). Metacorrection: Domain-aware meta loss correction for unsupervised domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3927-3936.
[14] Hoffman, J.. E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. Efros. and T. Darrell (2018). CyCADA: Cycle-consistent adversarial domain adaptation. Proceedings of the 35th International Conference on Machine Learning, Volume 80, pp. 1989-1998, PMLR.
[15] Jaderberg, M., K. Simonyan, A. Zisserman, and k. kavukcuoglu (2015). Spatial transformer networks. In Advances in Neural Information Processing Systems, Volume 28.
[16] Janssens, G., L. Jacques, J. Orban de Xivry, X. Geets, and B. Maeq (2011). Diffeomorphic registration of images with variable contrast enhancement. International Journal of Biomedical Imaging 2011.
[17] Kim, M. and H. Byun (2020). Learning texture invariant representation for domain adaptation of semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogmition (CVPR), pp. 12975-12984.
[18] Kingma, D. P. and J. Ba (2015). Adam: A method for stochastic optimization. In 3rd International Conference Learning Representations, ICLR 2015.
[19] Kumar, V. R., S. Yogamani, M. Bach, Witt, Milz, and P. Mäder (2020). Unrectdepthnet: Self-supervised monocular depth estimation using generic framework for handling common camera distortion models. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8177-8183 IEEE.
[20] Lee, S.-H. S.-K. Lee, and J.-S. Choi (2009). Correction of radial distortion using planar checkerboard pattern and its image. IEEE Transactions on Consumer Electronics 55(1), 27-33.
[21] Li, X., B. Zhang, P. V. Sander, and J. Liao (2019). Blind geometric distortion correction images through deep learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1885548664
[22] Li, Y., L. Yuan, and N. Vasconcelos (2019). Bidirectional learning for domain adaptation of semantic segmentation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6936 6945.
[23] Lin, C.-H. and S. Lucey (2017). Inverse compositional spatial transformer networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2568-2576.
[24] Long, M., Z. CAO.J. J. Wang, and M. I. Jordan (2018). Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems, Volume 31.
[25] Ma, H., X. Lin, Z. Wu, and Y. Yu (2021). Coarse-to-fing domain adaptive semantic segmentation with photometric alignment and category- regularization. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4051 4060.
[26] Mei, K., C. Zhu, J. Zou, and S. Zhang (2020). Instance adaptive self-training for unsupervised domain adaptation. In European Conference on Computer Vision (ECCV), pp. 415-430.
[27] Pan, F. I. Shin, F. Rameau, S. Lee, and I. S. Kweon (2020). Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. Proceedings of the IERE/VVF Conference on Computer Vision and Pattern Recognition (CVPR).
11

---

[28] Park, K., S. Woo, I. Shin, and I.-S. Kweon (2020). Discover, hallucinate, and adapt: Open com- pound domain adaptation for semantic segmentation. In 34th Conference on Neural Information Processing Systems, NeurlPS 2020.
[29] Richter, S. R., V. Vineet, S. Roth, and V. Koltun (2016). Playing for data: Ground truth from computer games. In European Conference on Computer Vision (ECCV), pp. 102-118. Springer.
[30] Ros, G., L. Sellart, Materzynska, D. Vazquez, and A. M. Lopez (2016). The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3234-3243,
[31] Selvaraju, R. R., M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra (2017), Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pp. 618 6-66..
[32] Sotiras, A., C. Davatzikos, and N. Paragios (2013). Deformable medical image registration: A survey. IEEE Transactions on Medical Imaging 32(7), 1153-1190.
[33] Tsai, Y.-H., W.-C. Wung. S. Schulter, K. Sohn, M.-H. Yang. and M. Chandraker (2018). Learning adapt structured output space for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7472-7481.
[34] Vu, T.-H., H. Jain, M. Bucher, M. Cord, and P. Pérez (2019). Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEER/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2517-2526.
[35] Wang. Z., M. Yu, Y. Wei, R. Feris, J. Xiong, W.-m. Hwu, T. S. Huang, and H. Shi (2020). Differential treatment for stuff and things: simple unsupervised domain adaptation method for semantic segmentation. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12635-12644.
[36] Yang, J., W. An, S. Wang, X. Zhu, C. Yan, and J. Huang (2020). Label-driven reconstruction for domain adaptation in semantic segmentation. In European Conference on Computer Vision (ECCV), pp. 480-498.
[37] Yang, Y. and S. Soatto (2020). Fda: Fourier domain adaptation for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4085-4095.
[38] Yogamani, S., C. Hughes, J. Horgan, G. Sistu, P. Varley, D. Dea, M. Uricár, S. Milz, M. Simon, K. Amende, al. (2019). Woodscape: A multi-task, multi-camera fisheye dataset for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9308-99 -91189
[39] Zhang, M. and P. T. Fletcher (2016). Bayesian statistical shape analysis on the manifold of diffeomorphisms. In Algorithmic Advances in Riemannian Geometry and Applications, pp. 1-23. Springer.
[40] Zhang, P., B. Zhang, T. Zhang, D. Chen, Y. Wang, and F. Wen (2021). Prototypical pseudo label denoising and target structure learning for domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 12414-12424.
[41] Zheng, Z. and Y. Yang (2021). Rectifying pseudo label lcarning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision 129(4). 1 1106-1122 1120.
[42] Zhu, J.-Y. Park, P. Isola, and A. A. Efros (2017). Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 2223-2232.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] See Section and Appendix A. We will also release FDD dataset upon acceptance.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.2 as well as Appendix A and B.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [Yes] See Appendix A.2.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets... (a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4 and Appendix A.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent obtained from people whose data you're you're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
13


Reviews:
[{"review_id": "RuxFDBQMHBh", "reviewer": "Reviewer_ihqK", "summary": "The author proposed a distortion-aware domain adaptation (DaDA) framework that is capable of modeling domain shifts in geometric deformation based on a relative distortion learning (RDL) method. \n\nThe proposed method tackles the task of unsupervised domain adaptation for semantic image segmentation where unknown optical distortion exists between the source and target images.\n\nAdequate experimental results also prove the validity of the proposed method。", "questions": "1.The figures in the text are far apart from where the figures are mentioned in the body text. It is suggested that the author shorten the corresponding distance to facilitate the reader’s comparative reading.\n\n2.Why are the edges of the reconstructed source domain image not neat? Such as the “d” part in figure2.\n\n3.In the \"Experimental Details\" section, the performance of both the “AdaptSeg+RA+RDL” and “AdvEnt+RA+RDL” methods decreases compared to the RA removal method when cityscapes is the source domain dataset and FDD is the target domain dataset. The authors need an explanation for this experimental result.\n\n4.Are the references in [17] and [18] duplicated?\n\n5.The format of references is not uniform. For example, the format of the reference in [28] differs from the rest of the literature.\n\n6.Most of the references are earlier than 2020. It is recommended to cite more papers in related fields after 2021.", "limitations": "The authors adequately address the limitations of their work and the potential negative social impact.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1.Strengths\nThis paper proposes a novel domain adaptation method for unsupervised semantic segmentation. And it combines geometric and optical distortion in domain shift. And the extensive experimental results highlight the effectiveness of our approach over state-of-the-art methods under unknown relative distortion across domains.\n2.Weaknesses\nSee the “Questions” part.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "RmBGanDRVtz", "reviewer": "Reviewer_QLQF", "summary": "This paper concerns the geometric distortion that causes domain gaps during unsupervised domain adaptation for semantic segmentation, which is motivated by the differences between rectilinear images and fisheye images. It is practically well-grounded.\nThe authors propose to train a deformation generator that consumes two images chosen from the source and target domains.\nThe output of the deformation generator is used to map the source image to the target style in terms of geometric distortion.\nThe training or adaptation is then happened by learning from the distorted source image and semantic segmentations.\nThe authors also propose a few benchmarks to evaluate the performance of the method and show good domain adaptation gain compared to baselines that concern global appearance change of the images.\nAn ablation study on each term in the proposed training loss is also performed.", "questions": "The main concerns are listed in the above section, please provide more information since they are related to the significance of the proposed problem and method.\n\nsome comments and questions on writting\n- the title \"DaDA: Distortion-aware Domain Adaptation for Unsupervised Semantic Segmentation\" may not be appropriate as semantic segmentation is not unsupervised.\n- ln 58 \"to enforce the semantic quality of relative deformation fields at the image- and the prediction level.\" how do you define semantic quality of deformation fields? also, what do you mean by \"image- and prediction level\"?\n- ln 142, the equation is confusing, phi_s2t is mapping from the source domain to the target domain, which takes pixel coordinates from source and maps it to source.", "limitations": "Did not observe much negative societal impact. On the limitation side, the authors indicate that other methods that deal with texture differences between domains can be combined with the proposed one to further improve performance. However, this is also an indicator that the paper with a clean motivation is evaluating on datasets with multiple causal factors, which does not convey a clean message.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- the motivation is practically meaningful and also the introduced setting is interesting.\n- the proposed method is simple yet shows good domain adaptation gain. Also, the proposed benchmarks could help develop more sophisticated methods that deal with both geometric distortion and appearance change.\n- the constraints to impose semantic consistency of the generated geometric distortion are interesting.\n- the paper is well-written and the idea is clearly conveyed.\n\nWeakness:\n- even though the motivation is well-grounded, the study of the validity of the problem, or how the motivating factor affects the adaptation performance is not crystal clear. For example, the authors should isolate geometric distortion from other factors like appearance and output space discrepancies. A simple experiment to do is to create a distorted target dataset from rectilinear images, and check how the other methods and the proposed method perform. The current message in this paper does not support a judgment on this aspect.\n- In theory, an ideal method should close the gap given only geometric distortion. But there is concern on whether the proposed method con do it under only geometric distortion or not, since the proposed randomly picks two images even though they are not in correspondence. This may be okay, since we can think of the proposed as a kind of spatial augmentation method, but we need to check the results.\n- The above concerns are critical for us to analyze the proposed method and the underlying problems that the proposed is trying to solve. For example, affine aug already works well on some of the directions in Table 1, does this mean that spatial augmentation/transformation is the key here? If so, would the proposed method a more sophisticated such augmentation method?\n- Also, how does the proposed method compare to spatial transformation generated by optical networks? The proposed do not really care about finding the correct underlying dense matching, which makes OF look like a good candidate to try, say, use OF to output some flow and then warp the images to the target domain or vice versa and then adapt? What is the difference then?\n- If camera distortion is the only factor that causes domain gap and is the focus of the current paper, then it seems that camera calibration would be a nice tool and can be performed quite well using existing tools. Then what does the paper tell us beyond that?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "AKd3LobaIMp", "reviewer": "Reviewer_83ri", "summary": "This paper proposes a novel segmentation framework that could be aware of distortion caused by optical and geometric reasons and thus improve the segmentation performance using unsupervised domain adaptation. The main contributions of this paper are (1) the framework that can be aware of distortion and utilize the information in the unsupervised semantic segmentation by using distortion-aware domain adaptation, (2) loss functions that effectively enforce the unsupervised domain adaptation and segmentation, and (3) new unsupervised domain adaptation benchmark including pre-existed datasets and newly proposed dataset, where the source and target images have additional domain gaps in the optical distortion.\n\nThe distortion-aware domain adaptation is realized by a novel Relative Distortion Learning (R.D.L.) using adversarial training (GAN) and Diffeomprohic transformation. The generator (G) generates deformation fields (visualization of distortion) between the source and target domain images. \n\nThe proposed method was evaluated using four datasets, especially two for the source domain, and the other two are utilized as the target domain. The images in the source domain are real-world rectilinear images, and the images in the target domains are optically and geometrically distorted images captured by fisheye images. Here, the authors propose a new dataset called the FDD captured by fisheye cameras (200-degree F.O.V.). \n\nThe experiments illustrate the outstanding performance of the proposed distortion-aware domain adaptation framework compared to other state-of-the-art deep learning models and other methods.\n", "questions": " * Questions & Discussions\n1. The definition of the domain adaptation should be more discussed. In this paper, the authors conducted the experiments with the environment where the rectilinear images are the source domain, whereas the images with extreme distortion are the target domain. However, in terms of “general” domain adaptation, the source and the target could be changed. Additionally, both the source domain and the target domain could contain the distortion on the edge. The reviewer is curious that the proposed framework is effective only in a limited environment. Alternatively, the reviewer is curious that other environments should not be discussed in the real-world image. If this case, the authors should clearly discuss this limitation. It is also discussed below [Limitations]-[Major issues].\n\n2. Effectiveness of the discriminator. The reviewer is curious about the effectiveness of the discriminator. Since the generator is well organized in the RDL and the loss function, the adversarial training can cause additional cost rather than a significant improvement in the accuracy. Otherwise, what if the discriminator categorizes the input images into domains (source and target). For instance, what if the discriminator aims to discriminate distortion style between I_T, I_(S->T), I_S, and I_(T->S). Since the generator generates the deformation field (S->T) and the inverse deformation field (T->S), training using four images can lead to the globally optimized discriminator, and it could be more efficient for the framework and the adversarial training. In addition, the ablation study for the discriminator could improve the quality of the manuscript.\n\n3. Cost of the proposed framework. The reviewer is curious about the training time, prediction time, and memory resources. In the manuscript, the authors addressed that the proposed framework is effective compared to the algorithm that removes the distortion from the image. Quantitative analysis or discussion can significantly improve the quality of the manuscript. For the reviewer, since the proposed network includes GAN-based architecture, segmentation-based architecture, and three or more loss functions, the training and prediction cost can be extremely burdened compared to the conventional algorithms.\n", "limitations": "1. Major issues\n- The field of interest suggested in this paper is narrow. The reviewers agree that the distortion can significantly degrade the segmentation performance when the image includes extreme distortion like the image captured using a fisheye camera. Even though real-world images contain the distortion, the reviewers could not be convinced that the minor distortion significantly degrades the segmentation performance. It could be justified through more experiments. The authors should significantly address the necessity of distortion-aware domain adaptation in the machine learning society instead of computer vision society. Additionally, the following concepts and experiments should be discussed to justify that the proposed framework is effective in the \"more general\" domain adaptive segmentation. (1) Utilization of the dataset containing distortion as the source domain and rectilinear images as the target domain. (2) Utilization of the dataset containing distortion as the source domain and another dataset with distortion as the target domain. Note that, this paper discussed the environment where the rectilinear images are utilized as the source domain and the images with distortion as the target domain. Or the authors should justify the experimental environment in this paper.\n\n- In addition, the above questions and the discussion are mainly concerned with a better quality of this paper for Neurips2022 (See Questions).\n\n2. Minor issues\n- Motivation for unsupervised learning needs to be more detailed. \nAs the authors illustrated in Line 45-47 on page 2, the motivation for unsupervised learning is the lack of a real-world public dataset with segmented annotations. Clearly, the lack of annotations for the dataset, including many images, could lead to unsupervised learning. However, that sentence author mentioned in the manuscript implies that the lack of the images (or dataset) leads the unsupervised learning in this paper. As the authors already knew, unsupervised learning with a few images could induce problems with biased training (e.g., overfitting). The reviewer expects that the precise motivation for unsupervised learning should be addressed strongly and clearly and that the authors can improve the quality of the manuscript with few modifications.\n\n- Small batch size should be discussed. As illustrated in Line 30-34, on page 1, in the supplementary file, the batch size for training deep learning models is four until convergence, and the batch size is significantly small. The importance of the normalization methods (e.g., batch normalization or group normalization) depends on the batch size. Therefore, discussing the batch size and the normalization methods would be better.\n\n- The vague description of the dataset and preparations should be improved. (1) the authors mentioned, \"Both source and target images are randomly cropped\" in the supplementary file in Line 32, on page 1. As the reviewer understood, the images in the target domain include the distortion on edge. At this moment, the random cropping method for the images in the target domain is nonsense. (2) In the experiments, the authors split images into training and validation sets with the random selection method. Is it for the cross-validation or hold-out methods? More explanations and statistical analysis (if authors conducted experiments with many folds) should be discussed even in the supplementary files.\n\n- Justification to search hyper-parameters should be discussed. Common hyper-parameters to train deep learning models are well illustrated in the manuscript and the supplementary files (especially the selection of beta 1-3). However, the reviewer is curious about the selection process of gamma. \n\n- (if possible) Experiments with other evaluation metrics could improve the quality of the manuscript. For instance, the boundary is a significantly important target in the segmentation tasks, and the distortion could degrade the boundary-oriented segmentation accuracy. Therefore, the discussion using evaluation metrics [1-3] for the boundary of the target objects can significantly improve the quality of the manuscript.\n\n[1] Fernandez-Moral, Eduardo, et al. \"A new metric for evaluating semantic segmentation: leveraging global and contour accuracy.\" 2018 IEEE intelligent vehicles symposium (iv). IEEE, 2018.\n\n[2] Lee, Kyungsu, et al. \"Boundary-oriented binary building segmentation model with two scheme learning for aerial images.\" IEEE Transactions on Geoscience and Remote Sensing 60 (2021): 1-17.\n\n[3] Cheng, Bowen, et al. \"Boundary IoU: Improving object-centric image segmentation evaluation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n3. Simple recommendations\n- Please re-check the typo and grammar errors to improve the quality of the manuscript.\n- Complicated quantitative analysis (Tables 5 and 6 in the supplementary material) should be clearly illustrated. \n- The reason why the authors illustrated the visualization of the class-activation map in the manuscript should be described in the main manuscript.\n- Public access to the code and the dataset should be preceded. ", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Strengths\n- The reviewer significantly understands the significance and importance of the task proposed in this paper. The distortion caused by the lens can significantly degrade the segmentation performance, especially the object on edge. In addition, domain adaptive semantic segmentation with lens distortion has rarely been studied to our best knowledge, such that the tasks proposed in this paper for the distortion-aware domain adaptation would be novel. Furthermore, since the dataset construction requires heavy cost, the construction of a large number of datasets for training machine learning models is noteworthy. \n\n- Additionally, the problem definition is clear. The authors defined the domain gap using not only distortion but also visual domain gap. At this point, in terms of the domain gap, the discrepancy between visual reasons and optical and geometric reasons should be discussed, and it should be decomposed. The authors proposed the “only” distortion-aware framework regardless of the visual domain gap.\n\n- The manuscript is well organized and well written.\n\n2. Weakness\n- More detailed descriptions are illustrated in the “Limitation” section. Please see below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept