Open Compound Domain Adaptation with Object Style Compensation for Semantic Segmentation
Authors: Tingliang Feng
Hao Shi
Xueyang Liu
Wei Feng
Liang Wan
Yanlin Zhou
Di Lin
Venue: NeurIPS 2023 poster (2023)
Keywords: Object Style Compensation
Open Compound Domain Adaptation
Semantic Segmentation
TL;DR: We propose the object style compensation, which is a novel open compound domain adaptation method for semantic segmentation.

Abstract: Many methods of semantic image segmentation have borrowed the success of open compound domain adaptation. They minimize the style gap between the images of source and target domains, more easily predicting the accurate pseudo annotations for target domain's images that train segmentation network. The existing methods globally adapt the scene style of the images, whereas the object styles of different categories or instances are adapted improperly. This paper proposes the Object Style Compensation, where we construct the Object-Level Discrepancy Memory with multiple sets of discrepancy features. The discrepancy features in a set capture the style changes of the same category's object instances adapted from target to source domains. We learn the discrepancy features from the images of source and target domains, storing the discrepancy features in memory. With this memory, we select appropriate discrepancy features for compensating the style information of the object instances of various categories, adapting the object styles to a unified style of source domain. Our method enables a more accurate computation of the pseudo annotations for target domain's images, thus yielding state-of-the-art results on different datasets.

Open Compound Domain Adaptation with Object Style Compensation for Semantic Segmentation
Tingliang Fengl.2 1.2
Hao Shi
Xueyang Liu'
Wei Feng
Liang Wan' Yanlin Zhou
Di Lin't
Cooleeee of Intelligence and Computing, Tianjin University
2 "Laboratory of Computation and Analytics Complex Management Systems (CACMS), Tianjin University
'Department of Automation, Tsinghua University *Dunhuang Academy
[fengtl, xyliu850569498, (raann@tju..eii edu. cn wfeng@ieee.org zhouyanl in@dha. ac.
shi thi-h220mails tsinghua. edu. cn Ande. www.1988ggain.coom
Abstract
Many methods of semantic image segmentation have borrowed the success of open compound domain adaptation. They minimize the style gap between the images source and target domains, more easily predicting the accurate pseudo annotations for target domain's images that train segmentation network. The existing methods style of the images, whereas the object styles of different
globally adapt the categories or instances are adapted improperly. This paper proposes the Object Style Compensation, where construct the Object-Level Discrepancy Memory with multiple sets of discrepancy features. The discrepancy features in a set capture the style changes of the same category's 's object instances adapted from target source domains. We learn the discrepancy features from the images of source and target domains, storing the discrepancy features in memory. With this memory, we select appropriate discrepancy features for compensating the style information of the object instances of various categories, adapting the object styles to a unified style of source domain. Our method enables more accurate computation of the pseudo annotations for target domain's 's images, thus yielding state-of-the-ard results on different datasets.
Introduction
The task of open compound domain adaptation (OCDA) for semantic segmentation aims to train the segmentation models n1 3 4
using source domain with pixel-level annotations and target domains mixed styles with no annotations. The model is trained to perform well on images from the target domains that share the same style seen during training, as well
5 6
open domain semantic
images that are not encountered during training. The recent methods 7 g 10 IT segmentation utilize the open compound domain adaptation (OCDA), which harnesses the annotated images captured in the open environments to train the segmentation
images and the annotation network. In this manner, the segmentation network learns from richer data with diverse object appearances while requiring reasonable effort for image labeling. Here, the annotated images belong to the source domain, while the annotation-free images are in the compound target domain for capturing the complexity of open environments.
There is gap between the image styles of the source and target domains, where the image styles usually are regarded as an array of scene-level properties (e.g., weather and lighting conditions). The
"Co-first authors. Equal contribution. Di Lin is the corresponding author of this paper.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Night Time Image
Night-to to tay Image
Day Time Images
(b)
Discrepancy Memorization
Style Compensation
OLDM
Set M
Set. M
Source Image
Target Image
aarget Image
Farget. Result
Set. M+
Source Image
aarge Image
Open Image
Open Result
Sytle Transfer
Compensatory Features
Feature Subtraction
Feature Addition
Figure 1: (a) Cars and pedestrians adapted from the night to day scenes. (b) The usual cars and pedestrians the real day scene. (c) OLDM contains the discrepancy features for representing
the difference of object styles in the source and target domains, which are used for adapting the object-level styles from target to source domains.
scene styles are adapted to narrow the gap between image styles, allowing the segmentation network to focus the intrinsic object appearances of every domain. It helps the network to predict accurate segmentation masks for many images in the compound target domain. The predicted masks play as the pseudo-pixel-wis annotations for tuning the segmentation network.
Typically, the adaptation of scene styles follows the homologous patterns to change the object styles in the same image. It lets various object categories undergo the style change along the same direction (see Figure [[a,, the clear pedestrians and overexposed cars adapted from the night to day scenes). In the common context, the objects in various categories, even different object instances in the same category, may have specific styles within a scene (see Figure T(b), the usual cars and pedestrians in the day time). Adapting usual context, leading to problematic annotations for the network training. Additional, The typical OCDA methods learn deep network parameters. The parameters learned from the source and target images in the training set. But they cannot trivially transform the open domain's object features that contain many unseen styles. Using the learned parameters directly may transform the object features of the open domain to the latent sub-space misaligned with the source domain.
styles may yield unreasonable object styles inconsistent with the
This paper advocates equipping OCDA with object style compensation for semantic segmentation. In contrast to the adaptation scene styles, the compensation respects the object style discrepancies captured for the independent
between the source and target domains. These discrepancies object categories and instances. Intuitively, the style discrepancies can be memorized during network learning. enables the appropriate style discrepancies, which can be regarded as the prior information, to be selected and added to the object features. Meanwhile, rather than using the network parameters transform the target/open domain's features the source domain, we learn the discrepancy features representing the style differences across domains. Consequently, we compensate the object features to adapt the object styles the similar style of source domain. The compensation yields consistent object context within the scene, helping the segmentation network to compute reliable pseudo annotations.
SSomooee performs the adaptation the image feature space. The latter is considered in this paper.

---

Specifically, we conduct the object style compensation for adapting the images from target to source domains. The pipeline comprises Discrepancy Memorization and Style Compensation, as illustrated in Sggure[cc. During Discrepancy Memorization, we construct a feature base, Object- Level Discrepancy Memory (OLDM), which consists of multiple feature sets. Each set contains the discrepancy features for the identical object category. Here, we compute the discrepancy feature by subtracting the target domain's 's object features from the same category's 's object features of the source domain, letting the discrepancy features represent the difference of the object styles across two domains. We compute the discrepancy features for the object instances in different images of target domain. During Style Compensation, given a query object in the target domain's 's image, select the discrepancy feature from the feature set of the requested category, where the discrepancy feature represents the instance much relevant to the query object. In this way. the discrepancy feature customizes the information, which is category- and instance-orientatee, for compensating the query object's feature. The compensated object features are used for regressing the pseudo annotations.
We conduct an intensive evaluation of the object style compensation. On the public datasets (e.g.. C-Driving 77, ACDC [12]. Cityscapes [13., KITTI [14), and WildDash (15)) that allows OCDA to assist semantic segmentation, the object style compensation surpasses state-of-the-art methods, demonstrating its effectiveness.
2 Related Work
2.1 Domain Adaptation for Semantic Segmentation
Numerous studies have been conducted in the realm of unsupervised domain adaptation (UDA), multi-target domain adaptation (MTDA), and domain generalization (DG) for the task of semantic segmentation. These investigations encompass plethora of approaches and methodologies, aimed at enhancing the performance and robustness of segmentation models.
the domain of UDA,
several notable works have contributed to this field [16 17
18
19
20
21
22
23
24
25 26 27).
These UDA methods encompass various strategies such as style regression 28 domain adversarial
training [18 29), and self-training []7 30 to adapt models to target domains. Similarly, the MTDA
domain has witnessed significant attention [3] 30 32 29 These studies tackle the challenge of adapting segmentation models to multiple target domains and employ diverse methodologies to achieve this objective. In contrast to UDA and MTDA, DG methods are characterized by their focus on learning Coomain-invariann representations. Researchers in this domain have explored techniques like learning domain invariant representation [33] 34 351, and data augmentation 36 37] 38]. to impart models with generalization capabilities across various domains. Turning to the specific context of OCDA, studies have employed diverse strategies, including style regression based on scene-level attributes 8 9 10, , and category-level attributes (7) to address the unique challenges posed by open
compound domain adaptation for segmentation task.
The existing methods adapt scene styles to fill in the gap of image appearances between different domains. They are insensitive to the object styles, which are critical to capture the style change the object appearances of various categories/instanees In contrast, we propose object-style compensation by respecting the individual categorics/instanees. Our approach appropriately adapts the object styles to the styles similar to the source domain and yield consistent context in the same scene.
2.2 Deep Network with Memorization for Visual Understanding
Recent studies demonstrate the importance of the deep network with memorization for visual
understanding [39 40 41 42 43
45 45 46. 47,
48
49
50
51
52
53 54, (55). Among
them, the application of memorization techniques has been notably observed within the realms of image processing [50] 52] 55 and video analysis 44 (56.. Furthermore, these methods can be categorized based on the level of granularity at which objects are stored in memory, with some
operating at the scene level 43 50 56]. while others are geared towards the category level 52 55)
The previous methods utilize memory that stores scene-level or category-level features extracted from the images. Because these features mainly capture a single domain's image style, they are less powerful for adapting broad range of image styles of the compounded target domain to the source domain. In contrast, we propose the external memory to store both category- and instance- level discrepancy features learned across different domains. These discrepancy features can directly
3

---

Source I.
Source
Source
Image
Encoder
Feature
Decoder Score Map
Discrepancy
Object-Level Discrepancy Memory
Style Compensation
Shared
Shared
Memorization
Target Image
Target
Tarret Score Map R, (a) Overview of the obj ect style compensation
Intermediate
Compensated Feature Map
Compensated Score Map
Feature
Fr
Segmentation Head
R.
Set. 1
F. Fa
A
Set
x A
Category-key
F. ()
Set Mr
Set K
A
F F2
Fz(z,
OLDM
(b) Discrepancy Memorization
(c) Style Compensation
O Category- Key
Object Features
Discrepancy
Representative Key Features
Simliarity Measurement
Features
Sets Selection
O
Features
Features
Update
Figure 2: (a) Overview of the object style compensation including discrepancy memorization, object- level discrepancy memory, and style compensation. (b) In the discrepancy memorization, we use the object features of the source and target domains, along with their differences. to update the category-key, representative-key, and discrepancy features in OLDM. (c) Given the object features of the target domain, we compute the similarities between the representative-key features and each object feature of the target domain, which is compensated by the discrepancy features weighted by the similarities.
compensate the images of the target domains, whose category- and instance-level styles adapted to the source domain. The alignment of the object styles different domains' images helps the segmentation network to focus the intrinsic object appearances for producing the pscudo labels.
Method Overview
We illustrate the object style compensation in Figure I,, I, E Hx are the images in the source and target domains. I,/I, is associated with/without the pixel-wise annotation. We denote Y. E #xxx L (L is the category number) as the ground-truth annotation for I,, The encoder extracts the object feature maps F. F, E R Bdxxxx xC from I,, I, C' indicates the feature channels. F,(,,)) Fr, y) E RC represent the object located (æ, y) in the source and target images.
The object style compensation includes Discrepancy Memorization and Style Compensation. The discrepancy memorization uses F., F, to construct the Object-Level Discrepancy Memory (OLDM). OLDM stores the discrepancy features, which capture the change of object styles from target to source domain. For the target image I, the style compensation uses discrepancy features to adapt the object style information of Fr, yielding compensated feature map F, E H used by decoder to compute the pseudo annotation Y, IR Xxx.
Discrepancy Memorization The discrepancy memorization only takes place during network training. Here, we construct category-key and OLDDM as {(A,, M,) A; E IR contains pairs of category-key feature (e.g., Ai) and feature set (e.g., M,). A, is the category- key feature that captures the source domain' 's representative style of the 1th category. The set Mi, = ((N i. D,,.) Ni, Dim E C.C m 1,... ...,,,, M] has pairs of representative-key feature (e.g., Ni.m.) and discrepancy feature (e.g., Dt.m). where m is the index of the representative-key feature. Ntm. captures representative instance-level style of the 1th category in target domain. We associate N,,m with the discrepancy feature D which captures the change of the representative instance-level style from target to source domain.
1, 1.... L.). which

---

We illustrate the discrepancy memorization in Figure 2(b), where we update the category-key representative-key. and discrepancy features. We use the 1t category's object features in F, to update the category-key feature A₁. In the set M, we measure the similarities between the 1 category's
object features in F, and the representative-key features {Nim m = 1, ...., M). According to these similarities, we use the 1t category's object feature Fx y) updates representative-Ke
features, while calculating the difference between A and F2 v) to update the discrepancy features (Dr.m m 1..., M).
Style Compensation We conduct the style compensation during training and testing. We illustrate the style compensation in Figure c). where we use the discrepancy features in OLDM to compensate the target image's object feature map F.. Given the object feature F, (2 y) as a query, we use segmentation head regress its category scores R₂(2,, y) R#xW*L selecting the feature sets [M,...., MR of OLDM for the style compensation. In Mr. we calculate the similarities between (r y) and the representative-
(N m 1, ..., M These similarities weight the discrepancy features [Da.m m 1, ..... M). After averaging the weighted discrepancy features of each set, the results are added Fz y). yielding the compensated feature (x y). Fe fed into the decoder for regressing the pseudo annotation Y. We borrow the pseudo annotation Y paired with the target image I to fine-tune the segmentation network.
Object Style Compensation
4.1 Discrepancy Memorization
During network training, we employ discrepancy memorization to construct the Object-Level Discrep- ancy Memory (OLDM). We denote category-key and OLDM as {(A,, Ml) A; E C , 1, ..., L}. where M2 = {N Dt.m) Ni,,; Drm E RC, m 1,.., M). We use the feature maps F,, F, E #x xC which are extracted from the source and target images I, I E RR x3 by encoder, update the category-key, representative-key. and discrepancy features in OLDM (see FigureB(b),
Update of Category-Key Features Given the feature map F, of the source image 1,, we update the category-key features { {A, E IC 1 ..., L}. We employ decoder to predict a category for every object feature in the map F.. The source image I, has the ground-truth annotation Y, E R #xxx The decoder predicts the category scores R. (z. y) E R', where R. v. E is the score for predicting the object located at (ar v) of the source image to be the 1th category. We assume the category leads to the highest score R, y, (). The predicted label is compared to the ground-truth label Y (x, y). The correct prediction means the object feature F, (2 y) is reliable for updating the category- key feature A₁ as:
A Ar + XF v), s.t. R. (x, 1, = (R, 1), ..., (2 y,, L)}. 1- Y,
where <- means the update by overwriting. À is ratio for controlling the information of F, (2, y) injected into A₁. In [T. A; aggregates the information of an array of object features, which consistently point to the 10 category. It helps A, to capture the representative object style of the 7th category in source domain, which is used to compute discrepancy features.
and Discrepancy Features We use the feature map F, of the target image I, to update the representative-key features in OLDM. For the object feature Fr(x, y) E RC, we use the intermediate segmentation head regress the category scores in R2(2,,) intermediate segmentation head takes input as the target feature for regressing the category score map, and R2 y. I) e R is the score for predicting the object located at (æ, y) of the target image to be the Ith category. We use F2 y) to update the representative-key feature N as:
Update of Representative-Kae
R2_ where
NIm Nu. + Wi.. Fem.Fela Fe v), Dim < Di.m + Wi. (A - F.(x,))) s.t. Re s.teyyll==axxx max
1), R L)} Mi Wi.m Ntm F (z.
(2)
VC
The category/ has the highest score in (R, y. 1), ..... (, We classify the object feature Fr Fr(z, y) to the eh category, thus updating the representative-key features (Ntm m 1,.., M
5

---

in the set M, for the same category. W.m E R is a weight, which is calculated as the dot-product between the representative-key feature Nm. and the object feature F Fz y), whose similarity measured. A high weight lets object feature contribute more information to representative-key feature.
Unlike decoder supervised by the ground-truth annotations of source images, the intermediate segmentation head is supervised by the pseudo annotations of target images. It may misclassify object feature, thus misleading the update of representative-key features. To remedy this issue, we set score threshold in Eq. (2). to select the reliable object feature Fz(2, y), whose highest score R2 (z. y. 1) is greater than y. This threshold only allows the object feature, whose category scores are confident, to update representative-kar features. Each representative-key feature aggregates the features of various object instances with similar styles.
In Eq 2 we also update the discrepancy features {Di. E R m== M). Here, we subtract the object feature (a y) from the category-key feature A,. Note that (r, y) and A extracted from the target and source images, respectively. Thus, the subtraction of these features captures the difference between the object styles of in target and source images, which is weighted by Wi,m to add to the discrepancy feature D Dt.m RCC
We remark that the update of OLDM is disabled during network testing. The style compensation uses the most updated category-key, representative-key, and discrepancy features in OLDM.
4.2 Style Compensation
The style compensation works during network training and testing. harnesses the category- and representative-ket features to find the appropriate discrepancy features in OLDM, which compensate for the object features of target image (see Figure c)).
Compensation of Object Features Given the object feature Fx y) query, which is extracted from the target image I, we use the intermediate segmentation head to predict the category scores R, (z, y)- Next, we select the representative-key and discrepancy features from {M,, Mxl }. where Mlç- = {(N
N Dk.m E RC m = I,..,,M. M to compensate the object feature F, (Z, v), yielding the compensated feature F, v) E RC as:
Fr(x,y) = Fe y) +
ΣΣ
Wk.m D Dams
k m
Nk.m Fr v)
s.t. R2(2, k) E max K (R4 (r, 1), 1),.... R2 (2, y. L)}: Wk.m
(3)
VC
where max means to select the top-K relevant categories. We use Eq. (3 to achieve the compensated
feature F. E "XXX which used for computing the pseudo annotation for the target image I,-
Computation of Pseudo Annotations Based on the compensated feature F,, the decoder of the segmentation network predicts the category score map R, E R We use the category score map R, to produce the pseudo annotation Y, E #x where the pixel-wise annotation Yx(x,)) y) E R for the pixel located (2,y) is determined as:
max R4 v, 1), Re(2, y, L)) R, > y, 1),
Yx,
0
max Re y, 1).... Re(x, y, L)} >
(4)
ignored 2 max R2 (x. v, 1), R2 (x,,,,
The pixel-wise annotation Y, y) a one-hot vector. We set the 1th channel Y, (z. y. to 1 (see the first case), when the category score R, y. () is higher than other scores in the set (Re(1, y, 1), Rx y, L)); otherwise, we set Yz(x, y, () (see the second case). It should be noted that the threshold is used in the third case, where too low scores let the pseudo annotations be ignored during network training. The computation of pseudo annotations only takes place during network training. For network testing, we resort to the category score map R, to predict the labels for all pixels, following the convention of semantic segmentation.

---

4.3 Supervision for Network Training
We use the ground-truth annotations source images and the pseudo annotations of target images to train segmentation network. We formulate the overall training objective L as:
L -= + Lpse-
(5)
gt and Lpse represent the objectives with the supervision of ground-trute and pseudo annotations.
Supervision of Ground-truth Annotations Based on the object feature map F, of the source image I.. decoder predicts the category score map Rg. We use the cross-entropy loss CE to penalize the difference between the predicted score map R., and the ground-truth annotation Y, as:
Ltt = CR,,, Y))-
Supervision of Pseudo annotations We use intermediate segmentation head to regress the category score map R, for the target image I, based the object feature map F,. We compute the cross- entropy loss, which penalizes the difference between the score map R. and the pseudo annotation Y, as the first term of the below Eq. (7)
Lpse CE(R, Y,) + CE(R,, Y.).
Moreover, we leverage the decoder to predict the category score map R,, based on the compensated
feature map F,, We again use cross-entropy loss (see the second term of Eq. ( to measure the segmentation errors in the score map R,, by comparing to the pseudo annotation Y,-
Experiments
5.1 Experimental Datasets
We use GTAS 577. SYNTHIA
58
C-
Table I: Divisions of the experimental datasets.
Driving 77 ACDC [12]. Cityscapes 13,
Train
Test
KITTI 14, and WildDash [15] datasets eval- uate our method. The images in a dataset may be subdivided into source, target, and open do- mains. All images with annotations in source domain and portion of images without anno- tations in target domain training. We evaluate the segmentation perfor- mances the rest images in target domain and all images in open domain. We list the division of these datasets in Table
Dataset
Total
Source
Target
Open
GTAS
24,966
24,966
SYNTHIA
9,400
9,400
C-Driving
16, 16,127
14,697
803
627
used for network
ACDC
1,906
1,200
306
400
Cityscapes
500
500
KITTT
200
200
WashDash
638
638
In the alation study of our method, we use 24,966 source images GTA5 dataset and 14,697 target images of C-Driving dataset for network training. 803 and 627 images of target and open domains in C-Driving dataset are used for testing. We report the segmentation performance regarding mean intersection-over-uniiee (mloUs) on target and open domains.
5.2
Implemenration Detail
We employ the PyTorch toolkit to implement segmentation network with Levect-Level Dis- crepancy Memory (OLDM). Our segmentation architecture is built upon DeepLab-V2 (59] with ResNet101 [60] backbone that has been pre-trained ImageNet [61]. To optimize the network, we utilize the SGD solver for 250,000 iterations. The initial learning rate is set to 0.00025, which undergoes linear decay throughout the training process. Each mini-batch consists of 4 images, comprising 2 source images and 2 target images. We apply random cropping, flipping, color jittering, and Gaussian blurring techniques to the training images, using crop size of 640x360 The network is trained on a single RTX 3090 GPU.
5.3 Ablation Study
Analysis of Diserepancy Memorization In Tables 2 and we study various strategies for updat- ing category-key. representative- and discrepancy features during the discrepancy memorization.
7

---

We report different strategies for using category- key features in 2 First, we disable the up- date of category-key features. This is done by re- moving OLDMM (see "w/o OLDM") during train- ing and testing, producing the performance of the stand-alone segmentation network. Another alternative uses the mean of the representative- key features in the same set to replace category- key feature but yields lower performances than our method (see "mean instances"). This is be-
Table 2: Results of various ways using category- key features. ml)U(T) and mloU(O) mean the mloUs on target and open domains.
Category-
Method
(T)U(T)
mloU(()
w/o OLDM
36.6
39.7
x
mcan instances
39.2
41.5
local
41.7
43.2
global
44.1
46.9
cause category-key and instance features are computed based on the image features of discrepant domains (i.e., source and target domains), making none of them replaceable. Second, we experiment with using the mean of the image features of source domain in local mini-batch to compute the category-key features, which are overridden by new mini-batch. This strategy achieves worse results than method (see "local"), because features globally (see "global"), which more comprehensively capture the object features of source domain.
different mini-batches compute category-
In Table we compare the performances of various strategies for using representative-key features. By eliminating the representative-kost features in OLDM, we lack the similarities be- tween the object features of target domain and the representative-key features for weighting the discrepancy features to compensate for object features. In this case, we experiment with sim- ply averaging discrepancy features (see "mean discrepancy"), using the category-level discrep- ancy rather than the instance-level, or directly computing the similarities between object features and discrepancy features (see "discrepancy simi- larity") for compensation. Without representative-ket features, these naive strategies lead to worse results than our method. This because the average discrepancy features and the alternative similari- ties inappropriately match object instances to the discrepancy features desired for compensation.
Table 3: Results of various ways of using representative-key features.
Rep.-Key
Method
mloU(T mloU((O)
mean discrepancy
37.8
38.8
x
category-leve discrepancy
38.1
40.2
discrepancy similarity
39.1
40.2
top-1 update
40.7
41.8
top-50% update
42.2
44.3
100% update
44.1
46.9
We also experiment with enabling and updating representative-ket features in various ways. For each set of representative-key and discrepancy features of the same category, we select the representative- key feature, which updated along with associated discrepancy feature by the object feature of target domain. It degrades the performances (see "top-1 update"), compared to adequately updating 50% or 100% of representative-key and discrepancy features (see "top-50% update" and "100% update")
In T we evaluate the performances of us- ing discrepancy features in different ways. Re- moving all discrepancy features in OLDM, we again degrade the whole pipeline to the back- bone segmentation network (see "w/o OLDM").
Table Results of various ways of using discrep- ancy features.
Discrepancy
Method
mloU(T)
mloU(O)
w/o OLDM
36.6
39.7
We also experiment with replacing discrep- ancy features with the discrepancy between the category- and representative-key features. This discrepancy only considers the difference be- tween representative features of source and tar- get domains. Our discrepancy features account for the differences between the representative features of source domain and various instances' object features of target domain. They show a stronger power for compensating the style information of relevant object instances. Moreover, we evaluate several alternatives for enabling discrepancy features for compensation. that differentiates discrepancy features according to categories and instances, an alternative method merges all discrepancy features as set (see "merged sets"). This method insensitive to the specific properties of categories and instances. Another way of separating discrepancy features into multiple sets is to use k-means clustering (see "multi-sets, k-means"), without depending on category-kos features. However, these methods neglect the useful category-key features, which select
key discrepancy
42.7
44.3
merged sets
39.6
41.5
multi-scts, k-means
41.7
43.3
multi-sets, category
44.1
46.9
contrast to our method
8

---

representative and discrepancy features for compensating the object features of the matched categories. Their performances lag behind our method (see "multi-sets, category").
Variants of Style Compensation In Tables and 6 evaluate the effectiveness of the style compensation by replacing it with other schemes during network testing. Here, OLDM has the optimized category-key, representative-key, and discrepancy features.
First, we remove the optimized OLDM and dis- able the style compensation. Another alterna- tive method of style compensation is averaging all of the discrepancy features without relying on the similarities between target images' and representative-key features for weighting dis- crepancy features. The above methods (see "w/o OLDM" and "mean discrepancy")) yield lower performances than method, which better utilizes discrepancy features for compensating the style information of different categories and instances in target domain.
Table 5: Results of various compensation methods.
Style Compensation
Method
mIoU(T) mloU(O)
x
w/o OLDM
36.6
39.7
discrepancy
40.7
41.9
instance similarity
44.1
46.9
Next, we analyze the quality of the pseudo an- notations computed by the style compensation. Without pseudo annotations, we only use the ground-truth annotations source images for training the segmentation network. We also eval- uate the quality of the pseudo annotations pro- duced by the intermediate segmentation head for supervising the segmentation network. With- out the accurate pseudo annotations computed based on compensated object features, these alternatives yield worse results than our method.
Table 6: Results of various ways of computing pseudo annotations.
Pseudo Annotations
Method
mloU(T)
mloU(O)
w/o pscudo
39.7
41.6
intermediate
42.4
44.3
final
44.1
46.9
5.4 External Comparison
In Table and we compare the performances of start-of-the-aa OCDA, UDA and DG methods 7 9 10
For fair comparison, have clearly specified the sets of images used for training and testing in each table. Our method of object style compensation outperforms an array start-of- the-art methods different datasets. In Figures and + compare the segmentation results of different methods, where our method yields better results.
62
6 Conclusion
Open compound domain adaptation has been successfully used to improve semantic image segmenta- tion performance. The popular methods globally adapt the scene styles of images. However, they unreasonably change the object styles of various categories and instances, forming unusual object
Table 7: Comparison with state-of-ther the-t-r methods. We clarify the source, target, and open domains for training and testing the compared methods at the top of table. mIoU11 and mloU16 mean the mloUs on Il and 16 categories, respectively.
Train:C- Drriingggt,,, Test:C- (riiC---iiiigT)
TrDin:ADDCT), Test: ACDC(T & O) SYNTHIA(S)
Method
Type
GTAS(S)
SYNTHIA(S)
GTAS(S)
mIoU(T)
'T)
mloU (T)
mloU(T)
mloU(O)
(T)
(6
Source-only
28.3
20.9
28.1
20.5
27.1
19.8
20.5
CDAS
OCDA
31.4
25.3
34.0
25.3
29.1
25.9
23.3
CSFU 9
OCDA
34,9
26.1
34.8
27.6
30.5
26.7
24.8
DACS 62
UDA
36.6
28.1
36.5
29.0
34.8
28.3
27.0
DHA 8
OCDA
37.1
29.9
37.6
29.5
37.5
29.2
27.3
AST
OCDA
38.8
31.1
38.9
30.7
39.2
30.1
27.9
ML.BPMM
OCDA
40.2
32.1
40.0
32.1
41.6
31.9
29.1
Ours
OCDA
44.1
35.6
43.7
35.7
44.1
34.7
36.4

---

Table Comparison with state-of-the-arr methods, CD, CS, KT. and WD mean the mloUs on the C-Driving, Cityscapes, KITTI, and WildDash datasets.
GTA5(S)
SYNTHIA(S)
Method
Тype
CD
CS
KT
WD
Avg
KT
WD
Avg
9SFU
OCDA
38.9
38.6
37.9
29.1
36.1
36.2
34,9
32.4
27.6
32.8
DACS 62
UDA
39.7
37.0
40.2
30.7
36.9
36.8
37.0
37.4
28.8
35.0
RobustNet 13
DG
38.1
38.3
40.5
30.8
37.0
37.1
38.3
40.1
29.6
36.3
DHA
OCDA
39.4
38,8
40.1
30.9
37.5
38.9
38.0
40.6
30.0
36,9
AST [O]
OCDA
40.7
40.3
41.9
32.2
38.8
40.5
39.8
41.6
30.7
38.2
ML- [I]
OCDA
42.5
41.7
44.3
34.6
40.8
42.6
41.1
43.4
30.9
39.5
Ours
OCDA
46.9
43.6
46.5
40.1
44.3
48.5
48.0
51.3
39.6
46.9
Impul
DHA [8]
AST [10]
ML-BPM[11]
Ours
Ground Truth
Figure 3: Segmentation results of different methods on the target domain of C-Driving.
Input
DHA [8]
AST [10]
ML-BPM[11]
Ours
Ground-Thi Truth
Figure 4: Segmentation results various methods on the open domains C-Driving and KITTI.
contexts in the adapted images along with incorrect pseudo annotations. In this paper, we propose novel idea of object-style compensation. Our method leverages the object-level discrepancy memory, where multiple sets of discrepancy features account for the style changes of objects in different categories. Furthermore, the discrepancy features in the same set individually consider instance-leve style changes, thus adapting the object styles finely. Our method enables a more accurate computation of the pseudo annotations of the images in the target domains, which eventually assists in training the segmentation network. the future, we will extend our method other applications (e.g., object detection and image restoration).
Negative Societal Impacts
Our approach facilitates the analysis of image semantics, rendering it applicable in various scenarios such as autonomous vehicles and image/video content comprehension. However, it important to exercise caution when utilizing the results, they may contain potentially problematic information. Mishandling such information could lead to infringements on privacy or economic interests.
Acknowledgments
We thank the anonymous reviewers for their constructive suggestions. This work is supported in parts by the National Key Research and Development Program of China (2020YFC152200) and the National Natural Science Foundation of China (62072334).
10

---

References
[1] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2881-2890 2017.
[2] Liang- Caing-hhihhhhe Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decode with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision, pages 801-818, 2018
[3] Di Lin, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Multi-scale context intertwining for semantic segmentation. In Proceedings of the European Conference on Computer Vision, pages 603-619, 2018.
[4] Di Lin, Dingguo Shen, Siting Shen, Yuanfeng Ji, Dani Lischinski, Daniel Cohen-Or, and Hui Huang. Zigzagnet: Fusing top-down and bottom-up context for object segmentation. In Proceedings of the IEED/CVF Conference Computer Vision and Puttern Recognition, pages 7490-7499, 2019.
[5] Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design semantic segmentation with transformers, Advances in Neural Information Processing Systems, 34: 34:12077-12090, 2021.
[6] Di Lin, Dingguo Shen, Yuanfeng Ji, Siting Shen, Mingrui Xie, Wei Feng, and Hui Huang. Tagnet: Learning configurable context pathways for semantic segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 45(2):2475 2022.
[7] Ziwei Liu, Zhongqi Miao, Xingang Pan, Xiaohang Zhan, Dahua Lin, Stella X Yu, and Boqing Gong. Open compound domain adaptation. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1 12406-12415 2020.
[8] Kwanyong Park, Sanghyun Woo, Inkyu Shin, and In Kweon. Discover, hallucinate, and adapt: Open compound domain adaptation for semantic segmentation. Advances in Neural Information Processing Systems, 33: 13:108699100680 10880, 2020.
[9] Rui Gong, Yuhua Chen, Danda Pani Paudel, Yawei Lá, Ajad Chhatkuli, Wen Li, Dengxin Dai, and Lue Van Gool. Cluster. split, fuse, and update: Meta-learning for open compound domain adaptive semantic segmentation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8344-8354, 2021.
[10] Jogendra Nath Kundu, Akshay R Kulkarni, Suvaansh Bhambri, Varun Jampani, and Venkatesh Babu Radhakrishnan. Amplitude spectrum transformation open compound domain adaptive semantic segmentation. In AAAI Conference on Artificial Intelligence, volume 36, pages 1220-1227,20 2022.
[11] Fei Pan, Sungsu Hur. Seokju Lee, Junsik Kim, and In So Kweon. MI-bpm: Multi teacher learning with bidirectional photometric mixing for open compound domain adaptation in semantic segmentation. In European Conference on Computer Vision, pages 236-251,220 2022.
[12] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Acde: The adverse conditions dataset with correspon- dences for semantic driving scene understanding. In IEEE/CVF International Conference Computer Vision, pages 10765- 10775, 2021.
[13] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3213 -2233, 2016.
[14] Hassan Abu Alhaija, Siva Karthik Mustikovela, Lars Mescheder, Andreas Geiger, and Carsten Rother. Augmented reality meets computer vision: Efficient data generation for urban driving scenes, International Journal of Computer Vision, (26:961-972 2018.
[15] Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger. and Gustavo Fernandez Dominguez. Wilddash Wilddash--ceaating hazard-aware benchmarks. In European Conference Computer Vision, pages 2018.
[16] Rui Li. Qianfen Jiao, Wenming Cao, Hau-San Wong, and Si Wu. Model adaptation: Unsupervised domain adaptation without source data. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2020.

---

[17] Jaemin Na, Heechul Jung, Hyung Jin Chang, and Wonjun Hwang. Fixbi: Bridging domain spaces for unsupervised domain adaptation. In JEEE/CVF Conference Computer Vision and Pattern Recognition. pages 1094-1103, 2021.
[18] Ni Xiao and Lei Zhang. Dynamic weighted learning for unsupervised domain adaptation. In IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 15242-15231 15251. 2021.
[19] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. Advances in Neural Information Processing Systems, 33: 33:16282-11 16222216292.2. 16292, 2020.
[20] Mengxue Li, Yi-Ming Zhai, You- Wei Luo, Peng-Fei Ge, and Chuan-Xian Ren. Enhanced transport distance for unsupervised domain adaptation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13936-139444 2020.
[21] Shuhao Cui, Shuhui Wang, Junbao Zhuo, Chi Su, Qingming Huang, and Qi Tian, Gradually vanishing bridge for adversarial domain adaptation. In /EEL/CVF Conference on Computer Vision and Pattern Recognition, pages 12455-12464, 2020.
[22] Fei Pan, Inkyu Shin, Francois Rameau, Seokju Lee, and In So Kweon. Unsupervised intra-domain adaptation for semantic segmentation through self-supervision. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3764-3773, 2020.
[23] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In AAAI Conference on Artificial Intelligence, volume 34, pages 6502-6509, 2020.
[24] Jogendra Nath Kundu, Naveen Venkat, R Venkatesh Babu, et al. Universal source- free domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4544 4553, 2020.
[25] Han-Kai Hsu, Chun-Han Yao, Yi-Hsuan Tsai, Wei-Chih Hung, Hung-Yu Tseng, Mancesh Singh, and Ming- Hsuan Yang. Progressive domain adaptation for object detection. IEEE/CVF Winter Conference on Applications of Computer Vision, pages 749 007997777220 2020.
[26] Yanchao Yang and Stefano Soatto. Fda: Fourier domain adaptation for semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4085-4095, 2020.
[27] Yunsheng Li, Lu Yuan, and Nuno Vasconcelos. Bidirectional learning for domain adaptation of semantic segmentation. In JEEE/CVF Conference on Computer Vision and Pallern Recognition, pages 6936-6945, 2019.
[28] Xinyi Wu, Zhenyao Wu. Yuhang Lu, Lili Ju, and Song Wang, Style mixing and patchwise prototypical matching for one-shot unsupervised domain adaptive semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence. volume 36, pages 2740-2749. 2022.
[29] Takashi Isobe. Xu Jia, Shuaijun Chen, Jianzhong He, Yongjie Shi, Jianzhuang Liu, Huchuan Lu, and Shengjin Wang. Multi-target domain adaptation with collaborative consistency learning. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8187-8196, 2021.
[30] Antoine Saporta, Tuan- Hung Vu, Matthieu Cord, and Patrick Pérez. Multi-target adversarial frameworks for domain adaptation in semantic segmentation. In IEEE/CVF International Conference Computer Vision, pages 9072 908 1, 2021.
[31] Behnam Gholami, Pritish Sahu, Ognjen Rudovic, Konstantinos Bousmalis, and Vladimir Pavlovic. Unsu- pervised multi- target domain adaptation: An information theoretic approach. IEEE Transactions on Image Processing, 29:3993 4002, 2020.
[32] Subhankar Roy, Evgeny Krivosheev, Zhun Zhong, Nicu Sebe, and Elisa Ricci. Curriculum graph co- teaching for multi-target domain adaptation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5351 53661, 2021.
[33] Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. Advances in Neural Information Processing Systems, 34:2427-2440, 2021.
[34] Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaoqing Liu, and Yizhou Yu. Compound domain gen- eralization via meta- knwweenn encoding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 71 1119-7129, 2022.
[35] Xufeng Yao, Yang Bai, Xinyun Zhang, Yuechen Zhang. Qi Sun, Ran Chen, Ruiyu Li, and Bei Yu. Pel: Proxy-based contrastive learning for domain generalization. In Proceedings of the IEEE/CVF Conference Computer Vision and Patern Recognition, pages 7097-7107, 2022.
12

---

[36] Zijian Wang, Yadan Luo, Ruihong Qiu, Zi Huang. and Mahsa Baktashmotlagh. Learning to diversify for single domain generalization. In IEEE/CVF International Conference on Computer Vision, pages 834-843, 2021.
[37] Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 34:20210- 14-20210-20229, 2021.
[38] Juwon Kang, Sohyun Lce, Namyup Kim, and Suha Kwak. Style neophile: Constantly secking novel styles for domain generalization, In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7130-7140, 2022.
[39] Yan Huang and Liang Wang- Acmm: Aligned cross-modal memory for few-shot image and sentence matching. In IEEE/CVF International Conference on Computer Vixion, pages 5774-5783, 2019.
[40] Marcella Coria,, Mattco Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578-10587, 2020.
[41] Minfeng Zhu, Pingbo Pan, Wei Chen, and Yi Yang. Dm-gan: Dynamic memory generative adversarial net- works for text-to-image synthesis. In JEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 5802-5810, 2019.
[42] Scoung Wug Oh, Joon- Joon-Young Lee, Ning Xu, and Scon Joo Kim. Vidco object segmentation using space- time memory networks. In JEEF/CVF International Conference on Computer Vision, pages 9226-9235, 2019.
[43] Huaibo Huang, Aijing Yu, and Ran He. Memory oriented transfer learning for semi-supervised image deraining. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7732-7741, 2021.
[44] Zhian Liu, Yongwei Nie, Chengjiang Long, Qing Zhang, and Guiqing Li. A hybrid video anomaly flow reconstruction and flow-guided frame prediction. In
detection framework via IEEE/CVF International Conference Computer Vision, pages 13588-13597, 2021.
[45] Hui Chen, Guiguang Ding, Xudong Liu, Zijia Lin, Ji Liu, and Jungong Han. Imram: Iterative matching with recurrent attention memory for cross-modal image-text retrieval. In JEFE/CVF Conference Computer Vision and Pattern Recognition, pages 12655-12663, 2020.
[46] Somi Jeong, Youngjung Kim, Eungbean Lec, and Kwanghoon Sohn. Memory-guided unsupervised image-to-image translation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6558-6567, 2021.
[47] Xin Feng, Wenjie Pei, Fengjun Li, Fanglin Chen, David Zhang, and Guangming Lu. Generative memory- guided semantie reasoning model for image inpainting. IEFF Transactions Circuits and Systems for Video Technology, 32(( 1):7432-7447, 2022.
[48] Kaiming Hc, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729-9738, 2020.
[49] Rui Xu, Minghao Guo, Jiaqi Wang, Xiaoxiao Li, Bolei Zhou, and Chen Change Loy. Texture memory- augmented deep patch-based image inpainting. IEEE Transactions on Image Processing, 30:91 12-9124, 2021.
[50] Tingliang Feng, Wei Feng, Weiqi Li, and Di Lin. Cross-image context for single image inpainting. Advances in Neural Information Processing Systems, 15:1474-1487, 2022.
[51] Wenguan Wang. Tianfei Zhou, Fisher Yu, Jifeng Dai, Ender Konukoglu. and Luc Van Gool. Exploring cross-image pixel contrast for semantic segmentation. In IREE/CVF International Conference on Computer Vision, pages 7303-7313, 2021.
[52] Jian Liang. Dapeng Hu, and Jiashi Feng. Domain adaptation with auxiliary target domain-oriented classifier. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -1642, 2021.
[53] Vibashan Vs. Vikram Gupta, Poojan Oza, Vishwanath A Sindagi, and Vishal M Patel. Mega-eda: Mem- ory guided attention for category- aware unsupervised domain adaptive object detection. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4516 456666422222 2021.
13

---

[54] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d++ denoised self-training for unsupervised domain adaptation 3d object detection. IEEE Trunsactions Puttern Analyxis and Machine Intelligence, 2022.
[55] Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn. Pin the memory: Learning generalize semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 1350-43660 2022.
[56] Ho Kei Cheng and Alexander Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. In European Conference on Computer Vision, pages 640-655 658. Springer, 2022.
[57] Stephan R Richter, Vibhav Vineet, Stefan Roth, Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pages 102-118, 18, 2016.
[58] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez. and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In IEEE Conference on Computer Vision and Pattern Recognition, pages
2016.
[59] Liang-Chieh Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions Pattern Analysis and Machine Intelligence, 40(4):834 848, 2017.
[60] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference Computer Vision and Pattern Recognition, pages 770-778, 2016.
[61] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In JEEE Conference on Computer Vision and Pattern Recognition, pages 248 255. leee, 2009.
[62] Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. Dacs: Domain adaptation via cross moos--dmminn mixed sampling. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1379 1389, 2021.
[63] Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne Kim. Seungryong Kim, Jaegul Choo. Robustnet: Improving domain generalization urban-scent segmentation via instance selective whitening. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11580-1 1580-11590, 2021.
14


Reviews:
[{"review_id": "f7SXUb9sJO", "reviewer": "Reviewer_LZYW", "summary": "This paper introduces a new method for OCDA called Object Style Compensation, which focuses on adapting the style changes of different categories or instances of objects rather than just the overall scene style. This stored information is used to select the appropriate discrepancy features for compensating the style information of object instances. Then, object features are compensated using selected discrepancy feature. The authors construct the Object-Level Discrepancy Memory by storing multiple sets of discrepancy features. The proposed method achieves SOTA on OCDA benchmarks.", "questions": "- The explanation on L110 indicates that 'l' represents the category, but there is no explanation about what 'm' signifies. Does 'm' mean the number of instances?\n- The quality of the pseudo labels seems likely to influence Discrepancy Memorization. Is there a guarantee of a certain level of quality even in the initial stages of learning?", "limitations": "There are no limitations to potential negative societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Using the style of objects is both ideal and intriguing.\n- Experiments show the effectiveness of the proposed method.", "weaknesses": "- How exactly are the objects obtained? It's understandable that categories can be distinguished by pseudo labels, but it's unclear how different instances within the same class are differentiated.\n- Why is storing style differences important? Are there any advantages to storing these discrepancies rather than just performing style regression?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">### 1. It's unclear how different instances within the same class are differentiated.\n\nThank you for pointing out this confusion between the instance-key feature and the object instance. In this paper, we do not need to differentiate the instances as those in the instance segmentation task. Please see the second part of “Response to Common Questions”. We plan to revise the “instance-key feature” to the “representative-key feature” to fix the inappropriate name.\n\n>### 2. Are there any advantages to storing these discrepancies rather than just performing style regression?\n\nWe respectfully point out that the style regression (e.g., ML-BPM [ECCV 2022], AST [AAAI 2022], CSFU [CVPR2021], DHA [NeurlPS2020], and CDAS [CVPR2020]) unsatisfactorily addresses Open Compound Domain Adaptation (OCDA), where the styles of different object categories and instances should be appropriately transformed. It motivates us to store the features of the style discrepancies, providing a more explicit and flexible way of changing the styles at the category and instance levels. Please see our discussion in the first part of “Response to Common Questions”. \n\n>### 3. There is no explanation about what “$m$” signifies.\n\nWe denote “$m=1,…,M$” as the index of the representative-key feature, where “$M$” is the total number of the representative styles of the instances in the same object category. “$M$” serves as a hyper-parameter. In Section 2.2 “Sensitivity Analysis of Memory Capacity” of the supplementary file, we have experimented with changing “M” and examining the effect on the segmentation performance. We will add the explanation to the main paper.\n\n>### 4. Is there a guarantee of a certain level of quality even in the initial stages of learning?\n\nTo reduce the negative impact of low-quality pseudo labels on the Discrepancy Memorization, we pre-train the backbone segmentation network on the source images with the ground-truth segmentation masks (see Section 1 of the supplementary file). The pre-trained network yields more reliable pseudo labels, stabilizing the Discrepancy Memorization even in the initial stage.\n\n>### 5. There are no limitations to potential negative societal impact.\n\nSection 4 of the Supplementary Materials addresses the negative societal impacts. Our approach fosters comprehensive image analysis. However, the analysis may contain problematic information, possibly leading to the infringement of economic interests."}, {"review_id": "yzGfKrshg3", "reviewer": "Reviewer_uGsB", "summary": "The authors propose a novel target-to-source feature style transfer approach for open compound domain adaptation. Inspired by the observation of the existence of object style discrepancy when converting a target image to a source style, they design Object-Level Discrepancy Memory (OLDM). This module saves the category-key features of the source domain, instance-key features of the target domain, and discrepancy features between source and target domains. When performing style transfer from target features to source features, they compensate for the object inconsistency using discrepancy features specific to each object. They evaluate their method on various driving datasets in semantic segmentation.", "questions": "As I mentioned in the weaknesses section, this work seems to be most related to multi-target domain adaptation. Could the authors explain this research from the perspective of domain generalization?\n\nWhile proposing source-to-target feature style transfer in addition could potentially offer performance advantages, what is the reason behind only proposing the target-to-source approach?\n\nThis method seems to require instance masks, but how do we obtain instance masks from a dataset that only has semantic masks?", "limitations": "They address the limitations of their work in supplementary materials.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Their motivation is good and intuitive. I agree that there will be a certain degree of appropriate object style for each domain.\n\nIt appears that well-designed modules are in place based on motivation.", "weaknesses": "In the related work section, there is a simple listing of various papers, and the flow of each research is not well organized. Furthermore, there is little mention of papers published after 2021, which seems to indicate a lack of investigation into recent papers.\n\nBased on the proposed method alone, it seems that it is not specifically designed for open compound domain adaptation but rather for multi-target domain adaptation. The difference between the two tasks lies in whether there is a consideration for how to generalize to unseen domains, which does not appear to be addressed in this paper. The proposed target-to-source feature style transfer method is suitable for multi-target domain adaptation as it aims to obtain more accurate pseudo labels for each target domain. [1] presents a concept of transferring source features into various styles to achieve domain generalization. By incorporating additional well-designed techniques such as source feature style randomization, it can be considered a suitable method for open compound domain adaptation.\n\nThis research focuses on semantic segmentation, but the method requires instance masks. However, the paper does not explain how to obtain instance masks.\n\nMinor issues.\n- The content in Figure 2 seems to be sufficiently understandable from Figure 1-(c). It would be better to combine Figures 2 and 3 appropriately and allocate more space to the related work section.\n- There are three duplicated reference numbers of '23' on line 65.\"\n\n\n[1] WildNet: Learning Domain Generalized Semantic Segmentation from the Wild (CVPR'22)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">### 1. In the related work section, there is a simple listing of various papers, and the flow of each research is not well organized. Furthermore, there is little mention of papers published after 2021, which seems to indicate a lack of investigation into recent papers.\n\nThanks. We have added 14 publications since 2021 to the related work. We plan to reorganize the related work as below.\n\nIn the flow of \"domain adaptation for semantic segmentation\", the unsupervised domain adaptation (UDA) and multi-target domain adaptation train networks through the style regression (SMPPM [AAAI2022]), domain adversarial (DWL [CVPR2021], CCL [CVPR2021]), and self-training (FixBi [CVPR2021], MTKT [ICCV2021]). In contrast, the domain generalization (DG) methods involve learning domain invariant representation (T3A [NeurIPS2021], COMEN [CVPR2022], PCL [CVPR2022]) and data augmentation (L2D[ICCV2021], MBDG[NeurIPS2021], StyleNeophile[CVPR2022]). Regarding various methodologies for open compound domain adaptation (OCDA), they conduct the style regression based on scene-level (DHA [NeurIPS2020], CSFU [CVPR2021], AST [AAAI2022], ML-BPM [ECCV2022]) and category-level (CDAS [CVPR2020]) attributes.\n\nIn the flow of \"deep network with memorization for visual understanding\", the memorization technique is demonstrated in the context of image (CICM [NeurIPS2022], ATDOC [CVPR2021], PinMemory [CVPR2022]) and video (HF2-VAD [ICCV2021], XMem [ECCV2022]) tasks. Additionally, based on the granularity at which objects are stored in memory, these methods can be categorized at either the scene level (MOSS [CVPR2021], CICM [NeurIPS2022], XMem [ECCV2022]) or the category level (ATDOC [CVPR2021], PinMemory[CVPR2022]).\n\n>### 2. The proposed method is not specifically designed for open compound domain adaptation but rather for multi-target domain adaptation. How to generalize to unseen domains is not addressed.\n\nOur method of style compensation is specifically designed to transform the object features of the target and open domains to the source domain. In the first part of “Response to Common Questions”, we explain why our method can be generalized better to address the image segmentation on the unseen domains than the popular methods. \n\n>### 3. WildNet presents a concept of transferring source features into various styles to achieve domain generalization. By incorporating additional well-designed techniques such as source feature style randomization, it can be considered a suitable method for open compound domain adaptation.\n\nThanks for pointing out this relevant work, which will be added to our paper for discussion. WildNet borrows the object features of the wild domain, which is usually provided by large-scale datasets (e.g., ImageNet), to enrich the object features of the source domain. WildNet extends the source domain, where the object features can cover the unseen domains. Though this extension is effective, it is infeasible to cover all unseen domains exhaustively.\n\nIn contrast to WildNet, we follow another research direction of addressing OCDA. We store the discrepancy features between the source and target domains for style compensation, which can be reasonably generalized to transform the object features of the open domain to the source domain. Note that these works along different directions can work together to achieve better performances. Intuitively, the object features extended by WildNet belong to the target domains newly created. We add these extended features to the object features in the target domains in the experimental datasets. Next, we trivially compute the discrepancy features based on the object features of the source and target domains, where the discrepancy features are used for compensating the features of the open domain. In the table below, we compare WildNet, style compensation, and their combination, where the last method achieves the best performances on different datasets.\n\n\n| method | type | C-Driving(Open) | ACDC(Open) | Cityscapes(Open) | KITTI(Open) | WildDash(Open) |\n| - | :-: | :-: | :-: | :-: | :-: | :-: |\n| WildNet | DG | 42.3 | 40.8 | 40.7 | 41.5 | 34.2 |\n| Ours | OCDA | 46.9 | 44.1 | 43.6 | 46.5 | 40.1 |\n| Ours+WildNet | OCDA | 48.3 | 45.8 | 44.1 | 47.4 | 42.7 |\n\n>### 4. This method seems to require instance masks, but how do we obtain instance masks from a dataset that only has semantic masks?\n\nWe clarify that instance masks are unnecessary in our method. Please see the second part of “Response to Common Questions”. We plan to revise the “instance-key feature” to the “representative-key feature” to fix the inappropriate name.\n\n>### 5. The content in Figure 2 seems to be sufficiently understandable from Figure 1(c). It would be better to combine Figures 2 and 3 appropriately and allocate more space to the related work section.\n\nThanks. Figure 1(c) shows the importance of considering the style difference between different object categories and instances. It provides a high-level motivation for proposing the style compensation of different categories and instances. But it contains little technical information. Thus we need Figures 2 and 3 to provide technical details, We agree that Figures 2 and 3, whose redundant elements can be trimmed to better understand the technical pipeline. Please see the revised and merged figures in Figure 2 of the rebuttal file.\n\n>### 6. There are three duplicated reference numbers of '23' on line 65.\"\n\nThanks. We have fixed the duplication.\n\n>### 7. What is the reason behind only proposing the target-to-source approach?\n\nWith the target-to-source compensation, we transform the object features of the target domains to have a similar style to the source domain. Consequently, the segmentation network only needs to learn from the visual styles of the source domain, producing reliable pseudo labels for the images of the target domains with more variant styles. This advantage cannot be achieved by conducting source-to-target compensation.\n"}, {"review_id": "JaZ6zuPb4b", "reviewer": "Reviewer_XmuL", "summary": "The paper introduces Object Style Compensation which involves constructing an Object-Level Discrepancy Memory consisting of multiple sets of discrepancy features to minimize the style differences between the source and target domains and ensure the styles of different object categories or instances within the scene to be adapted well. In particular, the paper learns these discrepancy features from images of both domains and stores them in memory. By leveraging this memory, appropriate discrepancy features are selected to compensate for the style information of object instances across various categories, aligning their styles to a unified style of the source domain. This method enables more accurate computation of pseudo annotations for target domain images and achieves state-of-the-art results on different datasets.", "questions": "1. What’s the runtime for the current method?\n\n2. What are the failure cases?\n", "limitations": "The author didn’t discuss the limitations of the current proposed method.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Overall the paper is well organized and easy to read.\n\n- The idea of using the object-level discrepancy memory for domain adaptation is interesting.\n\n- Achieved better semantic segmentation performance than SOTA methods.\n", "weaknesses": "- What's the value of the parameter $\\gamma$ set in Eqn (2) and (4), and how to determine its value? I would like to suggest the authors to run a group of experiments with different values of $\\gamma$ to see how the value impacts the final performance.\n\n- The implementation detail is missing in the main paper. I would like to suggest the authors move this part from the supplementary to the main paper so that the reader can well understand and even reproduce the results if necessary.\n\n- The current paper still misses one important baseline, i.e., using the category-level discrepancy rather than the instance-level.\n\n- For the visualization part, the visualization in the current paper is still very simple. I am curious whether the current method can handle the complicated scenes (e.g., with overlap and occlusion between objects with same or different categories).\n\n\n- The authors didn’t provide the failure cases and the corresponding analysis in the current paper. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">### 1. What's the value of the parameter $\\gamma$ set in Eqn (2) and (4), and how to determine its value? I would like to suggest the authors to run a group of experiments with different values of $\\gamma$ to see how the value impacts the final performance.\n\nThanks. We have provided this analysis in Section 2.2 “Sensitivity Analysis of score threshold \\gamma” of the supplementary material. Figures 3 and 4 of the supplemental material report the performances with different values of \\gamma on the C-Driving dataset. The $\\gamma$, which leads to the best performance on the C-Driving dataset, is finally used for evaluating our method on different datasets.\n\n>### 2. The implementation detail is missing in the main paper. I would like to suggest the authors move this part from the supplementary to the main paper so that the reader can well understand and even reproduce the results if necessary.\n\nThanks. We will move the necessary details to the main paper according to your suggestion.\n\n>### 3. The current paper still misses one important baseline, i.e., using the category-level discrepancy rather than the instance-level.\n\nThanks. The table below compares the performances achieved by the category- and level-level discrepancy features. Compared to the baseline without style compensation (“w/o OLDM”), the category-level discrepancy features (“category-level discrepancy”) help to transform the target and open domains’ object features to the source domain, finally yielding better performances. However, the category-level discrepancy features insufficiently account for the style difference between the object instances of the target/open and the source domains. Thus, their performances lag behind our method with the instance-level discrepancy features.\n\n| method | mIoU(Target) | mIoU(Open) |\n| :-: | :-: | :-: |\n| w/o OLDM | 36.6 | 39.7 |\n| category-level discrepancy | 38.1 | 40.2 |\n| Ours | 44.1 | 46.9 |\n|\n\n>### 4. For the visualization part, the visualization in the current paper is still very simple. I am curious whether the current method can handle the complicated scenes (e.g., with overlap and occlusion between objects with same or different categories).\n\nThanks. In Figure 3 of the rebuttal file, we zoom in on some regions from the segmentation results. Our method of object-level style compensation yields satisfactory segmentation results in complicated scenes where the objects are overlapped or occluded. We will add these results to the main paper and supplementary file.\n\n>### 5. The authors didn’t provide the failure cases and the corresponding analysis in the current paper.\n\nWe respectfully clarify that the discussion on the failure cases has been provided in Section 3.1 of the supplementary file. In this discussion, images with adverse weather conditions lead to unsatisfactory results. The failure cases also appear when the images of the open domains exhibit significant deviations from the source domain. These factors make it challenging to select the appropriate discrepancy features from OLDM. Though our method outperforms other methods, we will investigate how to reduce failure cases in challenging scenarios.\n\n>### 6. What’s the runtime for the current method?\n\nIn Figure 1(c) of the supplementary file, we study the testing time of our method by changing the capacity of OLDM. With the default capacity (M=50), we need 0.380 seconds on average to process a 1280x720 image. We also compare the testing time of our method with other methods in the table below. Our method achieves better performance at the cost of reasonable testing times (second per each 1280x720 image).\n\n| method | source-only | CDAS | CSFU | DACS | RobustNet | DHA | AST | ML-BPM | Ours |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| running time |0.345 |0.387 |0.383 |0.364 |0.378 |0.397 |0.411 |0.402 |0.380 |\n\n>### 7. The author didn’t discuss the limitations of the current proposed method.\n\nPlease see Section 3 of the supplementary file, where we have analyzed the limitations of our approach. This analysis comprises of two aspects: \"Failure Cases\" and \"Evaluation of OLDM in Cross-Dataset Scenarios\".\n"}, {"review_id": "6PMOdEsnWG", "reviewer": "Reviewer_okqW", "summary": "This paper deals with the open compound domain adaptive semantic segmentation problem. The motivation is to compensate the object style gap across domains to obtain more accurate pseudo labels for self-training. The main framework consists of two parts:  discrepancy memory and style compensation. For discrepancy memory, the aim is to memorize the category-level and instance-level feature discrepancy between target and source features. For style compensation, the aim is to compensate the style variation from source to target to obtain high-quality pseudo labels to facilitate the subsequent self-training procedure. Finally, two cross-entropy loss terms with ground-truth source annotations and target pseudo labels respectively are minimized to update the segmentation network. Experiment results show that the proposed method not only improves the segmentation performance on the target domain, but also yields better segmentation results on the open domain, compared to previous methods. ", "questions": "See the weakness part. ", "limitations": "I don't see serious societal issues. ", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The idea is simple and straightforward. \n- The paper is generally easy to follow and understand. \n- Ablations of various alternative solutions are conducted. ", "weaknesses": "- Some details are not clearly stated. For example, what is the intermediate segmentation head? From the start to the end of the paper, I don't find a formal definition of the open compound domain adaptive semantic segmentation task. A clear definition of the task may help the reader to better understand the proposed method. \n\n- The proposed method involves many hyper-parameters, e..g., the $\\lambda$ in Eq. (1); the $\\gamma$ in Eq. (2) and Eq. (4); the $K$ in Eq. (3), etc. As we know, for a domain adaptation problem, it is hard to perform the hyper-parameter selection. I just doubt how those hyper-parameters affect the final performance. The authors should provide empirical studies and analysis. \n\n-  For the Instance-Key and Discrepancy Features, they are instance-level (see Eq. (2)). I just doubt if such method can be scalable to large-scale datasets, as the $m$ may be super large (there are large numbers of pixels within one specific category). \n\n- For the ablation part, I suggest the authors use equations along with text to better describe each alternative solutions. Only with text descriptions, it is not easy for the reader to accurately capture the meaning that the authors want to express.\n\n- Some necessary ablations are missing. For example, in Eq. (7), how about moving the first term and just using the last term? In Eq. (2) and Eq. (3), how about not using the weighting mechanism, instead simply using the average? For Eq. (1), how about simply averaging features without $\\lambda$? \n\n- For Eq. (1), it actually may change the scale of feature values as time goes on. I just doubt why this could work. Maybe moving average is more suitable. \n\n- From the experiment results, I see the mIoU for the open domain is generally higher than the target domain, though open domain images are not utilized during training. Do the authors have any explanations? \n\n- From a high-level, I doubt if this way can really help the segmentation on the open domain, as the compensation is designed for the target, i.e., based on the distribution or statistics of target images or features. How can such compensation generalize to an unseen domain? \n\n- The paper should be polished further to make the descriptions more professional and accurate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">### 1. Some details are not stated, like the intermediate segmentation head and the definition of OCDA for semantic segmentation.\n\nFigure 2 of the rebuttal file illustrates the intermediate head as the convolutional layers. This head takes input as the target feature for regressing the category score map.\n\nThe existing works (i.e., ML-BPM [ECCV 2022], CSFU [CVPR2021], and DHA [NeurlPS2020]) have defined OCDA for semantic segmentation. In training, the segmentation network learns from the source domain, where the images have pixel-level annotations. The network also learns from the target images with different styles from the source images. The target images are given without annotations. Thus their pseudo annotations should be regressed for the network training. In testing, the segmentation network is evaluated on the open domain images with styles unseen in training. We will add the definition to the paper.\n\n>### 2. The method involves many hyper-parameters, which are hard to select. How do they affect the performance?\n\nIn Section 2.2 of the supplementary file, we have studied the hyper-parameters (i.e., the memory capacity $M$ in Eq. (2), the ratio $\\lambda$ in Eq. (1), the score threshold $\\gamma$ in Eqs. (2) and (4), the feature set number $K$ in Eq. (3)). The performances in Figures 1-5 of the supplementary file guide us to select the hyper-parameters, which leads to the best results on the C-Driving dataset are used for evaluating our method on the Cityscapes, KITTI, WildDash, and ACDC datasets. Though it is hard to select hyper-parameters, our approach with robustness uses the same set of hyper-parameters to outperform other methods on various datasets.\n\n>### 3. For the Instance-Key and Discrepancy Features, they are instance-level (see Eq. (2)). Can such a method be scalable to large-scale datasets, as the $m$ may be super large?\n\n“$m=1,…,M$” is the index of the instance-key feature, which captures one of the representative styles appearing with the same category's instances. “$M=50$” already leads to satisfactory performances on various datasets (see the second part of “Response to Common Questions”).\n\n>### 4. The authors should use equations with text to describe each ablation alternative.\n\nThanks. We plan to add the equations with Tables 2-6 as below.\n\nw/o OLDM: w/o Eqs. (1)(2)(3)\n\nglobal(T2), 100% update(T3), multi-sets, category(T4), instance similarity(T5), final(T6): w/ all Eqs. in paper\n\nT2\n\nmean instances$$A_l = \\frac{1}{M} \\sum_m N_{l,m}$$\nlocal$$A_l = \\frac{1}{bs} \\sum_{bs} F_s(x,y)$$\nT3\n\nmean discrepancy$$D_{l,m} \\leftarrow \\frac{N\\cdot D_{l,m} + (A_l - F_t(x,y))}{N+1},\\widetilde{F_t}(x,y) = F_t(x,y) + \\sum_k \\sum_mD_{k,m}$$\ndiscrepancy similarity$$w_{l,m} = \\frac{D_{l,m} \\cdot F_t(x,y)}{\\sqrt{C}}$$\ntop-1 update$$m = argmax \\\\{w_{l,m}\\\\}, w_{l,m}=\\frac{N_{l,m} \\cdot F_t(x,y)}{\\sqrt{C}}$$\ntop-50% update$$m\\in max_{\\frac{M}{2}}\\\\{w_{l,m}\\\\}, w_{l,m}~~as~above$$\nT4\n\nkey discrepancy$$D_{l,m} = A_l - N_{l,m}$$\nmerged sets$$A \\leftarrow A + \\lambda F_s(x,y),N_m \\leftarrow N_m + w_mF_t(x,y),D_m \\leftarrow D_m + w_m(A-F_t(x,y)),w_m = \\frac{N_{m} \\cdot F_t(x,y)}{\\sqrt{C}}$$\nmulti-sets, k-means$$C_i=\\\\{D_m | d(D_m, c_k)\\ge d(D_m,c_i)~~for~k\\neq i\\\\}, c_i=\\frac{1}{|C_i|}\\sum_{D_m\\in C_i}D_m$$\nT5\n\nmean discrepancy$$\\widetilde{F_t}(x,y) = F_t(x,y) + \\sum_mD_{k,m}$$\nT6\n\nw/o pseudo: w/o Eq. (4)\n\nintermediate: replace $\\widetilde{{\\bf R}}$ with ${\\bf R}$ in Eq. (4)\n\n>### 5. Ablations are missing. Eq. (7) w/o the first term? Eqs. (2-3) w/o weighting but the average? Eq. (1) w/o $\\lambda$ but the averaging?\n\nBy removing the first term of Eq. (7), we turn off the intermediate segmentation head in training, missing a chance to update the shallow object feature map $F_t$. It degrades the result on the C-Driving dataset from 44.1 to 38.4 mIoU.\n\nIn Eqs. (2-3), the weights measure the similarities between the object and instance-key features. They re-weight the discrepancy features to compensate for the object features of the target/open domain to the source domain. By replacing the weighting with the moving average, Eq. (3) fails to select the appropriate discrepancy features, degrading the result from 44.1 to 40.7 mIoU. Please also see the first part of “Response to Common Questions”.\n\nIn Eq. (1), we make little difference to the result by replacing $\\lambda$ with the moving average (44.1 v.s. 43.9 mIoU). Empirically, $\\lambda=0.001$ prevents the outlier object features of the source domain from primarily impacting the category-key features. It lets the category-key features converge faster than the moving average (250K v.s. 350K iterations).\n\n>### 6. Eq. (1) may change the scale of features as time goes on. Why this work? The moving average is more suitable.\n\nPlease see the last paragraph of our response to your fifth concern.\n\n>### 7. mIoU for open domain is generally higher than the target domain, though the open domain is not for training. Any explanation?\n\nWith the source domain of GTA5, better performances on the open domain of ACDC than the target domain have been found in the many methods (e.g., CDAS [CVPR 2020], AST [AAAI 2022], and ML-BPM [ECCV 2022] in Table 7(e)). Though this comparison between distinct domains with different images is unfair and out of the evaluation scope, this is partially because the open domain is near the source domain, where the features are easily transformed to. With the source domain of SYNTHIA far away from the open domains, this phenomenon is generally not found. Even so, our method excellently transforms the open object features to the source domain, yielding better performances than the target domain.\n\n>### 8. If this way can help the segmentation in the open domain, as the compensation is designed based on the distribution or statistics of target images or features? How can it generalize to an unseen domain?\n\nPlease see the second part of “Response to Common Questions”.\n\n>### 9. The paper should be polished.\n\nThanks. We will refine our paper."}, {"review_id": "NkAUTGeKdp", "reviewer": "Reviewer_CD2C", "summary": "This paper strives to capture various category-level object style information and then compensate the style information of the object instances from the target to the source domain in the open compound domain adaptation for semantic segmentation. And they evaluate the proposed method on various source and target datasets to verify the robustness and universality.", "questions": "See weaknesses. ", "limitations": "None", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Overall, the writing is well-organized and easy to follow. And the strengths can be summarized as follows:\n1. The proposed approach to handling the open compound domain adaptation task from a novel category-level style compensation perspective is interesting and will bring some insights to the community.\n2. The components of discrepancy memorization and style compensation are convincing. And the authors conducted comprehensive comparison experiments as well as ablation studies among various benchmarks.", "weaknesses": "There are some weaknesses listed as follows:\n1. You raise a valid point regarding the terminology \"Instance-Key\" which may not be appropriate in the context of semantic segmentation tasks where instance-level annotations are not available.\n2. You suggest merging the content in Table 7 or moving some sub-tables to the supplementary material to improve the readability and aesthetics of the paper's layout.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">### 1. You raise a valid point regarding the terminology \"Instance-Key\" which may not be appropriate in the context of semantic segmentation tasks where instance-level annotations are not available.\n\nPlease see our discussion in the second part of “Response to Common Questions”. The term “instance” of “instance-key feature” does not strictly point to an object instance. An instance-key feature is updated by the same category of object features, which are extracted from different images. The object features from different images naturally represent various object instances. Thus, the instance-key feature is named as such to indicate it is sensitive to the object instances. More specifically, the instance-key feature summarizes the same category’s object instances with similar visual styles. Consequently, an instance-key feature captures one of the representative styles, which widely appear with the object instances of the same category. We plan to revise the “instance-key feature” to the “representative-key feature” to fix the inappropriate name.\n\n>### 2. You suggest merging the content in Table 7 or moving some sub-tables to the supplementary material to improve the readability and aesthetics of the paper's layout.\n\nThanks for your valuable comment. As the below tables, we plan to merge Table 7(a-b), (c-d), (e-f), respectively, which hopefully improve the paper layout.\n\n**Table(a-b): Train: GTA5(Source)/SYNTHIA(Source), C-Driving(Target). Test: C-Driving(Target).**\n|||GTA5(S)|SYNTHIA(S)|SYNTHIA(S)|\n|----|:----:|:----:|:----:|:----:|\n|Method|Type|mIoU(T)|mIoU$^{16}$(T)|mIoU$^{11}$(T)|\n|Source-only|-|28.3|20.9|28.1|\n|CDAS|OCDA|31.4|25.3|34.0|\n|CSFU|OCDA|34.9|26.1|34.8|\n|DACS|UDA|36.6|28.1|36.5|\n|DHA|OCDA|37.1|29.9|37.6|\n|AST|OCDA|38.8|31.1|38.9|\n|ML-BPM|OCDA|40.2|32.1|40.0|\n|**Ours**|OCDA|**44.1**|**35.6**|**43.7**|\n\n**Table(c-d): Train: GTA5(Source)/SYNTHIA(Source), C-Driving(Target). Test: C-Driving(Open), Cityscapes(Open), KITTI(Open), WildDash(Open).**\n|||||GTA5(S)|||||SYNTHIA(S)|||\n|----|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|\n|Method|Type|CD|CS|KT|WD|Avg|CD|CS|KT|WD|Avg|\n|CSFU|OCDA|38.9|38.6|37.9|29.1|36.1|36.2|34.9|32.4|27.6|32.8|\n|DACS|UDA|39.7|37.0|40.2|30.7|36.9|36.8|37.0|37.4|28.8|35.0|\n|RobustNet|DG|38.1|38.3|40.5|30.8|37.0|37.1|38.3|40.1|29.6|36.3|\n|DHA|OCDA|39.4|38.8|40.1|30.9|37.5|38.9|38.0|40.6|30.0|36.9|\n|AST|OCDA|40.7|40.3|41.9|32.2|38.8|40.5|39.8|41.6|30.7|38.2|\n|ML-BPM|OCDA|42.5|41.7|44.3|34.6|40.8|42.6|41.1|43.4|30.9|39.5|\n|**Ours**|OCDA|**46.9**|**43.6**|**46.5**|**40.1**|**44.3**|**48.5**|**48.0**|**51.3**|**39.6**|**46.9**|\n\n**Table(e-f): Train: GTA5(Source)/SYNTHIA(Source), ACDC(Target). Test: ACDC(Target and Open).**\n|||GTA5(S)|GTA5(S)|SYNTHIA(S)|SYNTHIA(S)|\n|----|:----:|:----:|:----:|:----:|:----:|\n|Method|Type|mIoU(T)|mIoU(O)|mIoU$^{16}$(T)|mIoU$^{16}$(O)|\n|Source-only|-|20.5|27.1|19.8|20.5|\n|CDAS|OCDA|25.3|29.1|25.9|23.3|\n|CSFU|OCDA|27.6|30.5|26.7|24.8|\n|DACS|UDA|29.0|34.8|28.3|27.0|\n|DHA|OCDA|29.5|37.5|29.2|27.3|\n|AST|OCDA|30.7|39.2|30.1|27.9|\n|ML-BPM|OCDA|32.1|41.6|31.9|29.1|\n|**Ours**|OCDA|**35.7**|**44.1**|**34.7**|**36.4**|"}]

Author rebuttal:
## Response to Common Questions

>### **1. The advantage of style compensation based on the discrepancy features**
>- Reviewer LZYW-Q2 “Are there any advantages to storing these discrepancies rather than just performing style regression?”
>- Reviewer uGsB-Q2 “The proposed method is not specifically designed for open compound domain adaptation but rather for multi-target domain adaptation. How to generalize to unseen domains is not addressed.”
>- Reviewer okqW-Q8 “From a high-level, I doubt if this way can really help the segmentation on the open domain, as the compensation is designed for the target, i.e., based on the distribution or statistics of target images or features. How can such compensation generalize to an unseen domain?”

Below, we explain the advantage of our method of style compensation, which specifically transforms the object features of the target and open domains to the source domain. In Figure 1 of the rebuttal file, we conceptually compare the style compensation with the typical OCDA methods (e.g., ML-BPM [ECCV 2022], AST [AAAI 2022], CSFU [CVPR2021], DHA [NeurlPS2020], and CDAS [CVPR2020]) that optimize the parameters of the deep network to transform the object features of the source domain to the target domain, or vice versa.

*[Feature Transformation by Network Parameters]*

In Figure 1(a), we illustrate the source and target object features as the dots and crosses of different categories spanning over the latent space. The dot and cross of the same category represent distinct instances. The source and target features belong to separate clusters (the yellow as the source domain, the blue and red as the target domains) that indicate the source and target domains with distinct styles. The typical OCDA methods learn deep network parameters. The parameters (the dash arrows) transform the target object features into the source domain.

The parameters are learned from the source and target images in the training set. But they cannot trivially transform the open domain's object features  (see the dots and crosses in the green cluster) that contain many unseen styles. Using the learned parameters directly may transform the object features of the open domain to the latent sub-space misaligned with the source domain. A remedy is adaptively re-weighting and re-combining the learned parameters, which are generalized as the new parameters (denoted as the green arrow) to transform the open domain features. But the network contains many layers of parameters. How can the parameters correspond to the unknown number of target and open domains in different layers? How can we re-weight and re-combine these parameters without an explicit correspondence between the parameters and the target domains? These problems remain to be challenging. Moreover, we should not only transform the image-level features of the entire scenes but also the features of different object categories and instances, as we have discussed in the introduction of the main paper (see lines 30-36). This goal is hard to achieve without effectively generalizing the learned parameters.


*[Advantage of Style Compensation]*

Rather than using the network parameters to transform the target/open domain's features to the source domain, we learn the discrepancy features representing the style differences across domains. Figure 1(b) shows that we learn and store the discrepancy features. The discrepancy features (denoted as the solid blue and red arrows) explicitly correspond to different object categories and representative styles of instances. The stored discrepancy features are flexibly re-weighted and re-combined, based on the similarity between object features of target and open domains (see the weights $w_1$ and $w_2$ computed in Eq. (3)). In this way, we generalize the re-weighted discrepancy features. A new discrepancy feature (denoted as the green arrow) transforms the object features of the open domain to the source domain (see the formulation in Eq. (3) of the main paper). In Figure 1(c) of the rebuttal file, we visualize some of the object features of the open domain, which undergo style compensation or not. With the style compensation powered by the discrepancy features, we better transform the open domain’s object features into the source domain. In Table 7(c-f) of the main paper, we have evaluated our method on the target and open domains of different datasets, where our method outperforms other methods.


>### **2. Instance masks**
>- Reviewer LZYW-Q1 “It's unclear how different instances within the same class are differentiated.”
>- Reviewer uGsB-Q4 “The paper does not explain how to obtain instance masks.”
>- Reviewer okqW-Q3 “For the Instance-Key and Discrepancy Features, they are instance-level (see Eq. (2)). I just doubt if such method can be scalable to large-scale datasets, as the m may be super large (there are large numbers of pixels within one specific category).”
>- Reviewer CD2C-Q1 “You raise a valid point regarding the terminology Instance-Key which may not be appropriate in the context of semantic segmentation tasks where instance-level annotations are not available.”

We clarify that instance masks are unnecessary in our method. The term “instance” of “instance-key feature” does not strictly point to an object instance. An instance-key feature is updated by the same category of object features, which are extracted from different images. The object features from different images naturally represent various object instances. Thus, the instance-key feature is named as such to indicate it is sensitive to the object instances. More specifically, the instance-key feature summarizes the same category’s object instances with similar visual styles. Consequently, an instance-key feature captures one of the representative styles, which widely appear with the object instances of the same category. We plan to revise the “instance-key feature” to the “representative-key feature” to fix the inappropriate name.

Meta-review: Most reviewers generally found that the paper is well organized and easy to read; the proposed method is novel or interesting; and the proposed method is effective and achieves SOTA performance. However, there are two common questions: (1) How exactly are the object instances obtained, and what’s their influence on the scalability and limitation of the proposed method? (2) Why is storing style differences important, how can this method address the open compound domain adaptation problem, and how can it generalize to unseen domains? Authors provided detailed feedback on the two common questions. Specifically, for (1), they clarified that their method does not require object instance segmentation. Therefore, this question was due to unclear presentation of the paper and authors promised to modify the name of “instance” to “representation”. For (2), authors also provided a detailed explanation on the mechanism of the proposed method, which seems reasonable in addressing the generalization problem into adapting all features to source domain. There are also other concerns raised by reviewers, which have been carefully addressed by authors. After rebuttal, all ratings are positive (6, 6, 5, 5) except a negative score (3: Reject) by Reviewer okqW. Reviewer okqW provided a number of weakness points. Authors carefully addressed all these points. After rebuttal, Reviewer okqW provided feedback feeling still confused about the mechanism generalizing to unseen domains. AC assumes that except this concern, other concerns have been addressed. Authors further provided an explanation, which seems reasonable in addressing the concern. Considering all the above, the paper’s strengths overwhelm its weaknesses, and it shares some useful insights to the community. AC therefore is happy to accept the paper. Authors are required to address all reviewer comments and incorporate the rebuttal and discussion material to the camera-ready version of the paper.
Decision: Accept (poster)