SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbf {Y} _ {t} (x, y, l) = \left\{ \begin{array}{l l} 1 & \max  \{\widetilde {\mathbf {R}} _ {t} (x, y, 1), \dots , \widetilde {\mathbf {R}} _ {t} (x, y, L) \} = \widetilde {\mathbf {R}} _ {t} (x, y, l) > \gamma , \\ 0 & \max  \{\widetilde {\mathbf {R}} _ {t} (x, y, 1), \dots , \widetilde {\mathbf {R}} _ {t} (x, y, L) \} > \max  \{\widetilde {\mathbf {R}} _ {t} (x, y, l), \gamma \}, \\ \text {i g n o r e d} & \gamma \geq \max  \{\widetilde {\mathbf {R}} _ {t} (x, y, 1), \dots , \widetilde {\mathbf {R}} _ {t} (x, y, L) \}. \end{array} \right. \tag {4}
$$
The pixel-wise annotation  $\mathbf{Y}_t(x,y)$  is a one-hot vector. We set the  $l^{th}$  channel  $\mathbf{Y}_t(x,y,l)$  to 1 (see the first case), when the category score  $\widetilde{\mathbf{R}}_t(x,y,l)$  is higher than other scores in the set  $\{\widetilde{\mathbf{R}}_t(x,y,1),\dots,\widetilde{\mathbf{R}}_t(x,y,L)\}$ ; otherwise, we set  $\mathbf{Y}_t(x,y,l)$  to 0 (see the second case). It should be noted that the threshold  $\gamma$  is used in the third case, where too low scores let the pseudo annotations be ignored during network training. The computation of pseudo annotations only takes place during network training. For network testing, we resort to the category score map  $\widetilde{\mathbf{R}}_t$  to predict the labels for all pixels, following the convention of semantic segmentation.
4.3 Supervision for Network Training
We use the ground-truth annotations of source images and the pseudo annotations of target images to train segmentation network. We formulate the overall training objective  $\mathcal{L}$  as:
$$
\mathcal {L} = \mathcal {L} _ {g t} + \mathcal {L} _ {p s e}. \tag {5}
$$
$\mathcal{L}_{gt}$  and  $\mathcal{L}_{pse}$  represent the objectives with the supervision of ground-truth and pseudo annotations.
Supervision of Ground-truth Annotations Based on the object feature map  $\mathbf{F}_s$  of the source image  $\mathbf{I}_s$ , decoder predicts the category score map  $\mathbf{R}_s$ . We use the cross-entropy loss  $\mathcal{C}\mathcal{E}$  to penalize the difference between the predicted score map  $\mathbf{R}_s$  and the ground-truth annotation  $\mathbf{Y}_s$  as:
$$
\mathcal {L} _ {g t} = \mathcal {C E} \left(\mathbf {R} _ {s}, \mathbf {Y} _ {s}\right). \tag {6}
$$
Supervision of Pseudo annotations We use intermediate segmentation head to regress the category score map  $\mathbf{R}_t$  for the target image  $\mathbf{I}_t$ , based on the object feature map  $\mathbf{F}_t$ . We compute the cross-entropy loss, which penalizes the difference between the score map  $\mathbf{R}_t$  and the pseudo annotation  $\mathbf{Y}_t$  as the first term of the below Eq. (7).
$$
\mathcal {L} _ {p s e} = \mathcal {C E} \left(\mathbf {R} _ {t}, \mathbf {Y} _ {t}\right) + \mathcal {C E} \left(\tilde {\mathbf {R}} _ {t}, \mathbf {Y} _ {t}\right). \tag {7}
$$
Moreover, we leverage the decoder to predict the category score map  $\widetilde{\mathbf{R}}_t$ , based on the compensated feature map  $\widetilde{\mathbf{F}}_t$ . We again use cross-entropy loss (see the second term of Eq. (7)) to measure the segmentation errors in the score map  $\widetilde{\mathbf{R}}_t$ , by comparing to the pseudo annotation  $\mathbf{Y}_t$ .
5.1 Experimental Datasets
We use GTA5 [57], SYNTHIA [58], C-Driving [7], ACDC [12], Cityscapes [13], KITTI [14], and WildDash [15] datasets to evaluate our method. The images in a dataset may be subdivided into source, target, and open domains. All images with annotations in source domain and a portion of images without annotations in target domain are used for network training. We evaluate the segmentation performances on the rest images in target domain and all images in open domain. We list the division of these datasets in Table 1.
Table 1: Divisions of the experimental datasets.
Dataset	Total	Train	Test
Source	Target	Open
GTA5	24,966	24,966	-	-	-
SYNTHIA	9,400	9,400	-	-	-
C-Driving	16,127	-	14,697	803	627
ACDC	1,906	-	1,200	306	400
Cityscapes	500	-	-	-	500
KITTI	200	-	-	-	200
WashDash	638	-	-	-	638
In the alation study of our method, we use 24,966 source images of GTA5 dataset and 14,697 target images of C-Driving dataset for network training. 803 and 627 images of target and open domains in C-Driving dataset are used for testing. We report the segmentation performance regarding mean intersection-over-unions (mIoUs) on target and open domains.
5.2 Implementenration Detail
We employ the PyTorch toolkit to implement our segmentation network with Object-Level Discrepancy Memory (OLDM). Our segmentation architecture is built upon DeepLab-V2 [59] with ResNet101 [60] backbone that has been pre-trained on ImageNet [61]. To optimize the network, we utilize the SGD solver for 250,000 iterations. The initial learning rate is set to 0.00025, which undergoes a linear decay throughout the training process. Each mini-batch consists of 4 images, comprising 2 source images and 2 target images. We apply random cropping, flipping, color jittering, and Gaussian blurring techniques to the training images, using a crop size of  $640 \times 360$ . The network is trained on a single RTX 3090 GPU.
Analysis of Discrepancy Memorization In Tables 2, 3, and 4, we study various strategies for updating category-key, representative-key, and discrepancy features during the discrepancy memorization.
We report different strategies for using category-key features in Table 2. First, we disable the update of category-key features. This is done by removing OLDM (see "w/o OLDM") during training and testing, producing the performance of the stand-alone segmentation network. Another alternative uses the mean of the representative-key features in the same set to replace category-key feature but yields lower performances than our method (see "mean instances"). This is be
Table 2: Results of various ways of using categorykey features. mIoU(T) and mIoU(O) mean the mIoUs on target and open domains.
Category-Key	Method	mIoU(T)	mIoU(O)
x	w/o OLDM	36.6	39.7
mean instances	39.2	41.5
✓	local	41.7	43.2
global	44.1	46.9
cause category-key and instance features are computed based on the image features of discrepant domains (i.e., source and target domains), making none of them replaceable. Second, we experiment with using the mean of the image features of source domain in a local mini-batch to compute the category-key features, which are overridden by new mini-batch. This strategy achieves worse results than our method (see "local"), because we use different mini-batches to compute category-key features globally (see "global"), which more comprehensively capture the object features of source domain.
In Table 3, we compare the performances of various strategies for using representative-key features. By eliminating the representative-key features in OLDM, we lack the similarities between the object features of target domain and the representative-key features for weighting the discrepancy features to compensate for object features. In this case, we experiment with simply averaging discrepancy features (see "mean discrepancy"), using the category-level discrepancy rather than the instance-level, or directly
Table 3: Results of various ways of using representative-key features.
Rep.-Key	Method	mIoU(T)	mIoU(O)
x	mean discrepancy	37.8	38.8
category-level discrepancy	38.1	40.2
discrepancy similarity	39.1	40.2
✓	top-1 update	40.7	41.8
top-50% update	42.2	44.3
100% update	44.1	46.9
computing the similarities between object features and discrepancy features (see "discrepancy similarity") for compensation. Without representative-key features, these naive strategies lead to worse results than our method. This is because the average discrepancy features and the alternative similarities inappropriately match object instances to the discrepancy features desired for compensation.
We also experiment with enabling and updating representative-key features in various ways. For each set of representative-key and discrepancy features of the same category, we select the representative-key feature, which is updated along with associated discrepancy feature by the object feature of target domain. It degrades the performances (see "top-1 update"), compared to adequately updating  $50\%$  or  $100\%$  of representative-key and discrepancy features (see "top-  $50\%$  update" and "100% update").
In Table 4, we evaluate the performances of using discrepancy features in different ways. Removing all discrepancy features in OLDM, we again degrade the whole pipeline to the backbone segmentation network (see "w/o OLDM"). We also experiment with replacing discrepancy features with the discrepancy between the category- and representative-key features. This discrepancy only considers the difference between representative features of source and tar
Table 4: Results of various ways of using discrepancy features.
Discrepancy	Method	mIoU(T)	mIoU(O)
×	w/o OLDM	36.6	39.7
key discrepancy	42.7	44.3
✓	merged sets	39.6	41.5
multi-sets, k-means	41.7	43.3
multi-sets, category	44.1	46.9
get domains. Our discrepancy features account for the differences between the representative features of source domain and various instances' object features of target domain. They show a stronger power for compensating the style information of relevant object instances. Moreover, we evaluate several alternatives for enabling discrepancy features for compensation. In contrast to our method that differentiates discrepancy features according to categories and instances, an alternative method merges all discrepancy features as a set (see "merged sets"). This method is insensitive to the specific properties of categories and instances. Another way of separating discrepancy features into multiple sets is to use k-means clustering (see "multi-sets, k-means"), without depending on category-key features. However, these methods neglect the useful category-key features, which select
representative-key and discrepancy features for compensating the object features of the matched categories. Their performances lag behind our method (see "multi-sets, category").
Variants of Style Compensation In Tables 5 and 6, we evaluate the effectiveness of the style compensation by replacing it with other schemes during network testing. Here, OLDM has the optimized category-key, representative-key, and discrepancy features.
First, we remove the optimized OLDM and disable the style compensation. Another alternative method of style compensation is averaging all of the discrepancy features without relying on the similarities between target images' and representative-key features for weighting discrepancy features. The above methods (see "w/o OLDM" and "mean discrepancy") yield lower
Table 5: Results of various compensation methods.
Style Compensation	Method	mIoU(T)	mIoU(O)
X	w/o OLDM	36.6	39.7
✓	mean discrepancy	40.7	41.9
instance similarity	44.1	46.9
performances than our method, which better utilizes discrepancy features for compensating the style information of different categories and instances in target domain.
Next, we analyze the quality of the pseudo annotations computed by the style compensation. Without pseudo annotations, we only use the ground-truth annotations of source images for training the segmentation network. We also evaluate the quality of the pseudo annotations produced by the intermediate segmentation head for supervising the segmentation network. Without the accurate pseudo annotations computed
Table 6: Results of various ways of computing pseudo annotations.
Pseudo Annotations	Method	mIoU(T)	mIoU(O)
X	w/o pseudo	39.7	41.6
✓	intermediate	42.4	44.3
final	44.1	46.9
based on compensated object features, these alternatives yield worse results than our method.
5.4 External Comparison
In Table 7 and 8, we compare the performances of start-of-the-art OCDA, UDA and DG methods [7, 8, 9, 10, 11, 62]. For a fair comparison, we have clearly specified the sets of images used for training and testing in each table. Our method of object style compensation outperforms an array of start-of-the-art methods on different datasets. In Figures 3 and 4, we compare the segmentation results of different methods, where our method yields better results.
Open compound domain adaptation has been successfully used to improve semantic image segmentation performance. The popular methods globally adapt the scene styles of images. However, they unreasonably change the object styles of various categories and instances, forming unusual object
Table 7: Comparison with state-of-the-art methods. We clarify the source, target, and open domains for training and testing the compared methods at the top of table.  $\mathrm{mIoU}^{11}$  and  $\mathrm{mIoU}^{16}$  mean the mIoUs on 11 and 16 categories, respectively.
Method	Type	Train:C-Driving(T). Test:C-Driving(T)	Train:ACDC(T). Test: ACDC(T & O)
GTA5(S)	SYNTHIA(S)	GTA5(S)	SYNTHIA(S)
mIoU(T)	mIoU16(T)	mIoU11(T)	mIoU(T)	mIoU(O)	mIoU16(T)	mIoU16(O)
Source-only	-	28.3	20.9	28.1	20.5	27.1	19.8	20.5
CDAS[7]	OCDA	31.4	25.3	34.0	25.3	29.1	25.9	23.3
CSFU[9]	OCDA	34.9	26.1	34.8	27.6	30.5	26.7	24.8
DACS[62]	UDA	36.6	28.1	36.5	29.0	34.8	28.3	27.0
DHA[8]	OCDA	37.1	29.9	37.6	29.5	37.5	29.2	27.3
AST[10]	OCDA	38.8	31.1	38.9	30.7	39.2	30.1	27.9
ML-BPM[11]	OCDA	40.2	32.1	40.0	32.1	41.6	31.9	29.1
Ours	OCDA	44.1	35.6	43.7	35.7	44.1	34.7	36.4
Table 8: Comparison with state-of-the-art methods. CD, CS, KT, and WD mean the mIoUs on the C-Driving, Cityscapes, KITTI, and WildDash datasets.
Method	Type	GTA5(S)	SYNTHIA(S)
CD	CS	KT	WD	Avg	CD	CS	KT	WD	Avg
CSFU[9]	OCDA	38.9	38.6	37.9	29.1	36.1	36.2	34.9	32.4	27.6	32.8
DACS[62]	UDA	39.7	37.0	40.2	30.7	36.9	36.8	37.0	37.4	28.8	35.0
RobustNet[63]	DG	38.1	38.3	40.5	30.8	37.0	37.1	38.3	40.1	29.6	36.3
DHA[8]	OCDA	39.4	38.8	40.1	30.9	37.5	38.9	38.0	40.6	30.0	36.9
AST[10]	OCDA	40.7	40.3	41.9	32.2	38.8	40.5	39.8	41.6	30.7	38.2
ML-BPM[11]	OCDA	42.5	41.7	44.3	34.6	40.8	42.6	41.1	43.4	30.9	39.5
Ours	OCDA	46.9	43.6	46.5	40.1	44.3	48.5	48.0	51.3	39.6	46.9
Input
DHA [8]
AST [10]
ML-BPM [11]
Ours
Ground-Truth
Figure 3: Segmentation results of different methods on the target domain of C-Driving.
Figure 4: Segmentation results of various methods on the open domains of C-Driving and KITTI.
contexts in the adapted images along with incorrect pseudo annotations. In this paper, we propose a novel idea of object-style compensation. Our method leverages the object-level discrepancy memory, where multiple sets of discrepancy features account for the style changes of objects in different categories. Furthermore, the discrepancy features in the same set individually consider instance-level style changes, thus adapting the object styles finely. Our method enables a more accurate computation of the pseudo annotations of the images in the target domains, which eventually assists in training the segmentation network. In the future, we will extend our method to other applications (e.g., object detection and image restoration).
Negative Societal Impacts
Our approach facilitates the analysis of image semantics, rendering it applicable in various scenarios such as autonomous vehicles and image/video content comprehension. However, it is important to exercise caution when utilizing the results, as they may contain potentially problematic information. Mishandling such information could lead to infringements on privacy or economic interests.
We thank the anonymous reviewers for their constructive suggestions. This work is supported in parts by the National Key Research and Development Program of China (2020YFC1522701) and the National Natural Science Foundation of China (62072334).
[54] Jihan Yang, Shaoshuai Shi, Zhe Wang, Hongsheng Li, and Xiaojuan Qi. St3d++: denoised self-training for unsupervised domain adaptation on 3d object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[55] Jin Kim, Jiyoung Lee, Jungin Park, Dongbo Min, and Kwanghoon Sohn. Pin the memory: Learning to generalize semantic segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4350-4360, 2022.
[56] Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model. In European Conference on Computer Vision, pages 640-658. Springer, 2022.
[57] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European Conference on Computer Vision, pages 102-118, 2016.
[58] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In IEEE Conference on Computer Vision and Pattern Recognition, pages 3234-3243, 2016.
[59] Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4):834-848, 2017.
[60] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[61] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. IEEE, 2009.
[62] Wilhelm Tranheden, Viktor Olsson, Juliano Pinto, and Lennart Svensson. Dacs: Domain adaptation via cross-domain mixed sampling. In IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1379-1389, 2021.
[63] Sungha Choi, Sanghun Jung, Huiwon Yun, Joanne T Kim, Seungryong Kim, and Jaegul Choo. Robustnet: Improving domain generalization in urban-scene segmentation via instance selective whitening. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11580-11590, 2021.