SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tingliang Feng $^{1,2*}$  Hao Shi $^{1,3*}$  Xueyang Liu $^{1}$  Wei Feng $^{1,2}$
Liang Wan $^{1}$  Yanlin Zhou $^{4}$  Di Lin $^{1\dagger}$
<sup>1</sup>College of Intelligence and Computing, Tianjin University
$^{2}$ Laboratory of Computation and Analytics of Complex Management Systems (CACMS), Tianjin University
$^{3}$ Department of Automation, Tsinghua University  $^{4}$ Dunhuang Academy
{fengtl, xyliu850569498, lwan}@tju.edu.cn shi-h23@mails.tsinghua.edu.cn
wfeng@ieee.org zhouyanlin@dha.ac.cn Ande.lin1988@gmail.com
Many methods of semantic image segmentation have borrowed the success of open compound domain adaptation. They minimize the style gap between the images of source and target domains, more easily predicting the accurate pseudo annotations for target domain's images that train segmentation network. The existing methods globally adapt the scene style of the images, whereas the object styles of different categories or instances are adapted improperly. This paper proposes the Object Style Compensation, where we construct the Object-Level Discrepancy Memory with multiple sets of discrepancy features. The discrepancy features in a set capture the style changes of the same category's object instances adapted from target to source domains. We learn the discrepancy features from the images of source and target domains, storing the discrepancy features in memory. With this memory, we select appropriate discrepancy features for compensating the style information of the object instances of various categories, adapting the object styles to a unified style of source domain. Our method enables a more accurate computation of the pseudo annotations for target domain's images, thus yielding state-of-the-art results on different datasets.
The task of open compound domain adaptation (OCDA) for semantic segmentation aims to train the segmentation models [1, 2, 3, 4, 5, 6] using a source domain with pixel-level annotations and target domains of mixed styles with no annotations. The model is trained to perform well on images from the target domains that share the same style as seen during training, as well as on open domain images that are not encountered during training. The recent methods [7, 8, 9, 10, 11] of semantic segmentation utilize the open compound domain adaptation (OCDA), which harnesses the annotated images and the annotation-free images captured in the open environments to train the segmentation network. In this manner, the segmentation network learns from richer data with diverse object appearances while requiring reasonable effort for image labeling. Here, the annotated images belong to the source domain, while the annotation-free images are in the compound target domain for capturing the complexity of open environments.
There is a gap between the image styles of the source and target domains, where the image styles usually are regarded as an array of scene-level properties (e.g., weather and lighting conditions). The
*Co-first authors. Equal contribution.
$\dagger$  Di Lin is the corresponding author of this paper.
(b)
Figure 1: (a) Cars and pedestrians adapted from the night to day scenes. (b) The usual cars and pedestrians in the real day scene. (c) OLDM contains the discrepancy features for representing the difference of object styles in the source and target domains, which are used for adapting the object-level styles from target to source domains.
scene styles are adapted<sup>3</sup> to narrow the gap between image styles, allowing the segmentation network to focus on the intrinsic object appearances of every domain. It helps the network to predict accurate segmentation masks for many images in the compound target domain. The predicted masks play as the pseudo-pixel-wise annotations for tuning the segmentation network.
Typically, the adaptation of scene styles follows the homologous patterns to change the object styles in the same image. It lets various object categories undergo the style change along the same direction (see Figure 1(a), the clear pedestrians and overexposed cars adapted from the night to day scenes). In the common context, the objects in various categories, even different object instances in the same category, may have specific styles within a scene (see Figure 1(b), the usual cars and pedestrians in the day time). Adapting scene styles may yield unreasonable object styles inconsistent with the usual context, leading to problematic annotations for the network training. Additional, The typical OCDA methods learn deep network parameters. The parameters are learned from the source and target images in the training set. But they cannot trivially transform the open domain's object features that contain many unseen styles. Using the learned parameters directly may transform the object features of the open domain to the latent sub-space misaligned with the source domain.
This paper advocates equipping OCDA with object style compensation for semantic segmentation. In contrast to the adaptation of scene styles, the compensation respects the object style discrepancies between the source and target domains. These discrepancies are captured for the independent object categories and instances. Intuitively, the style discrepancies can be memorized during network learning. It enables the appropriate style discrepancies, which can be regarded as the prior information, to be selected and added to the object features. Meanwhile, rather than using the network parameters to transform the target/open domain's features to the source domain, we learn the discrepancy features representing the style differences across domains. Consequently, we compensate the object features to adapt the object styles to the similar style of source domain. The compensation yields consistent object context within the scene, helping the segmentation network to compute reliable pseudo annotations.
Someone performs the adaptation in the image or feature space. The latter one is considered in this paper.
Specifically, we conduct the object style compensation for adapting the images from target to source domains. The pipeline comprises Discrepancy Memorization and Style Compensation, as illustrated in Figure 1(c). During Discrepancy Memorization, we construct a feature base, Object-Level Discrepancy Memory (OLDM), which consists of multiple feature sets. Each set contains the discrepancy features for the identical object category. Here, we compute the discrepancy feature by subtracting the target domain's object features from the same category's object features of the source domain, letting the discrepancy features represent the difference of the object styles across two domains. We compute the discrepancy features for the object instances in different images of target domain. During Style Compensation, given a query object in the target domain's image, we select the discrepancy feature from the feature set of the requested category, where the discrepancy feature represents the instance much relevant to the query object. In this way, the discrepancy feature customizes the information, which is category- and instance-orientated, for compensating the query object's feature. The compensated object features are used for regressing the pseudo annotations.
We conduct an intensive evaluation of the object style compensation. On the public datasets (e.g., C-Driving [7], ACDC [12], Cityscapes [13], KITTI [14], and WildDash [15]) that allows OCDA to assist semantic segmentation, the object style compensation surpasses state-of-the-art methods, demonstrating its effectiveness.
2.1 Domain Adaptation for Semantic Segmentation
Numerous studies have been conducted in the realm of unsupervised domain adaptation (UDA), multi-target domain adaptation (MTDA), and domain generalization (DG) for the task of semantic segmentation. These investigations encompass a plethora of approaches and methodologies, aimed at enhancing the performance and robustness of segmentation models. In the domain of UDA, several notable works have contributed to this field [16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27]. These UDA methods encompass various strategies such as style regression [28], domain adversarial training [18, 29], and self-training [17, 30] to adapt models to target domains. Similarly, the MTDA domain has witnessed significant attention [31, 30, 32, 29]. These studies tackle the challenge of adapting segmentation models to multiple target domains and employ diverse methodologies to achieve this objective. In contrast to UDA and MTDA, DG methods are characterized by their focus on learning domain-invariant representations. Researchers in this domain have explored techniques like learning domain invariant representation [33, 34, 35], and data augmentation[36, 37, 38], to impart models with generalization capabilities across various domains. Turning to the specific context of OCDA, studies have employed diverse strategies, including style regression based on scene-level attributes [8, 9, 10], and category-level attributes [7] to address the unique challenges posed by open compound domain adaptation for segmentation task.
The existing methods adapt scene styles to fill in the gap of image appearances between different domains. They are insensitive to the object styles, which are critical to capture the style change of the object appearances of various categories/instances. In contrast, we propose object-style compensation by respecting the individual categories/instances. Our approach appropriately adapts the object styles to the styles similar to the source domain and yield consistent context in the same scene.
2.2 Deep Network with Memorization for Visual Understanding
Recent studies demonstrate the importance of the deep network with memorization for visual understanding [39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55]. Among them, the application of memorization techniques has been notably observed within the realms of image processing [50, 52, 55] and video analysis [44, 56]. Furthermore, these methods can be categorized based on the level of granularity at which objects are stored in memory, with some operating at the scene level [43, 50, 56], while others are geared towards the category level [52, 55].
The previous methods utilize memory that stores scene-level or category-level features extracted from the images. Because these features mainly capture a single domain's image style, they are less powerful for adapting a broad range of image styles of the compounded target domain to the source domain. In contrast, we propose the external memory to store both category- and instance-level discrepancy features learned across different domains. These discrepancy features can directly
(a) Overview of the object style compensation
(b) Discrepancy Memorization
(c) Style Compensation
Figure 2: (a) Overview of the object style compensation including discrepancy memorization, object-level discrepancy memory, and style compensation. (b) In the discrepancy memorization, we use the object features of the source and target domains, along with their differences, to update the category-key, representative-key, and discrepancy features in OLDM. (c) Given the object features of the target domain, we compute the similarities between the representative-key features and each object feature of the target domain, which is compensated by the discrepancy features weighted by the similarities.
compensate the images of the target domains, whose category- and instance-level styles are adapted to the source domain. The alignment of the object styles in different domains' images helps the segmentation network to focus on the intrinsic object appearances for producing the pseudo labels.
3 Method Overview
We illustrate the object style compensation in Figure 2(a).  $\mathbf{I}_s, \mathbf{I}_t \in \mathbb{R}^{H \times W \times 3}$  are the images in the source and target domains.  $\mathbf{I}_s / \mathbf{I}_t$  is associated with/without the pixel-wise annotation. We denote  $\mathbf{Y}_s \in \mathbb{R}^{H \times W \times L}$  ( $L$  is the category number) as the ground-truth annotation for  $\mathbf{I}_s$ . The encoder extracts the object feature maps  $\mathbf{F}_s, \mathbf{F}_t \in \mathbb{R}^{H \times W \times C}$  from  $\mathbf{I}_s, \mathbf{I}_t$ .  $C$  indicates the feature channels.  $\mathbf{F}_s(x,y), \mathbf{F}_t(x,y) \in \mathbb{R}^C$  represent the object located at  $(x,y)$  in the source and target images.
The object style compensation includes Discrepancy Memorization and Style Compensation. The discrepancy memorization uses  $\mathbf{F}_s, \mathbf{F}_t$  to construct the Object-Level Discrepancy Memory (OLDM). OLDM stores the discrepancy features, which capture the change of object styles from target to source domain. For the target image  $\mathbf{I}_t$ , the style compensation uses discrepancy features to adapt the object style information of  $\mathbf{F}_t$ , yielding the compensated feature map  $\widetilde{\mathbf{F}}_t \in \mathbb{R}^{H \times W \times C}$  used by decoder to compute the pseudo annotation  $\mathbf{Y}_t \in \mathbb{R}^{H \times W \times L}$ .
Discrepancy Memorization The discrepancy memorization only takes place during network training. Here, we construct category-key and OLDM as  $\{(\mathbf{A}_l,\mathbb{M}_l)\mid \mathbf{A}_l\in \mathbb{R}^C,l = 1,\dots,L\}$ , which contains pairs of category-key feature (e.g.,  $\mathbf{A}_l$ ) and feature set (e.g.,  $\mathbb{M}_l$ ).  $\mathbf{A}_l$  is the category-key feature that captures the source domain's representative style of the  $l^{th}$  category. The set  $\mathbb{M}_l = \{(\mathbf{N}_{l,m},\mathbf{D}_{l,m})\mid \mathbf{N}_{l,m},\mathbf{D}_{l,m}\in \mathbb{R}^C,m = 1,\dots,M\}$  has pairs of representative-key feature (e.g.,  $\mathbf{N}_{l,m}$ ) and discrepancy feature (e.g.,  $\mathbf{D}_{l,m}$ ), where  $m$  is the index of the representative-key feature.  $\mathbf{N}_{l,m}$  captures a representative instance-level style of the  $l^{th}$  category in target domain. We associate  $\mathbf{N}_{l,m}$  with the discrepancy feature  $\mathbf{D}_{l,m}$ , which captures the change of the representative instance-level style from target to source domain.
We illustrate the discrepancy memorization in Figure 2(b), where we update the category-key, representative-key, and discrepancy features. We use the  $l^{th}$  category's object features in  $\mathbf{F}_s$  to update the category-key feature  $\mathbf{A}_l$ . In the set  $\mathbb{M}_l$ , we measure the similarities between the  $l^{th}$  category's object features in  $\mathbf{F}_t$  and the representative-key features  $\{\mathbf{N}_{l,m} \mid m = 1,\dots,M\}$ . According to these similarities, we use the  $l^{th}$  category's object feature  $\mathbf{F}_t(x,y)$  updates representative-key features, while calculating the difference between  $\mathbf{A}_l$  and  $\mathbf{F}_t(x,y)$  to update the discrepancy features  $\{\mathbf{D}_{l,m} \mid m = 1,\dots,M\}$ .
Style Compensation We conduct the style compensation during training and testing. We illustrate the style compensation in Figure 2(c), where we use the discrepancy features in OLDM to compensate the target image's object feature map  $\mathbf{F}_t$ . Given the object feature  $\mathbf{F}_t(x,y)$  as a query, we use a segmentation head to regress its category scores  $\mathbf{R}_t(x,y) \in \mathbb{R}^{H \times W \times L}$ , selecting the feature sets  $\{\mathbb{M}_1, \dots, \mathbb{M}_K\}$  of OLDM for the style compensation. In  $\mathbb{M}_k$ , we calculate the similarities between  $\mathbf{F}_t(x,y)$  and the representative-keys  $\{\mathbf{N}_{k,m} \mid m = 1, \dots, M\}$ . These similarities weight the discrepancy features  $\{\mathbf{D}_{k,m} \mid m = 1, \dots, M\}$ . After averaging the weighted discrepancy features of each set, the results are added to  $\mathbf{F}_t(x,y)$ , yielding the compensated feature  $\widetilde{\mathbf{F}}_t(x,y)$ .  $\widetilde{\mathbf{F}}_t$  is fed into the decoder for regressing the pseudo annotation  $\mathbf{Y}_t$ . We borrow the pseudo annotation  $\mathbf{Y}_t$  paired with the target image  $\mathbf{I}_t$  to fine-tune the segmentation network.
4 Object Style Compensation
4.1 Discrepancy Memorization
During network training, we employ discrepancy memorization to construct the Object-Level Discrepancy Memory (OLDM). We denote category-key and OLDM as  $\{(\mathbf{A}_l,\mathbb{M}_l)\mid \mathbf{A}_l\in \mathbb{R}^C,l = 1,\dots ,L\}$  where  $\mathbb{M}_l = \{\left(\mathbf{N}_{l,m},\mathbf{D}_{l,m}\right)\mid \mathbf{N}_{l,m},\mathbf{D}_{l,m}\in \mathbb{R}^C,m = 1,\dots ,M\}$ . We use the feature maps  $\mathbf{F}_s,\mathbf{F}_t\in \mathbb{R}^{H\times W\times C}$ , which are extracted from the source and target images  $\mathbf{I}_s,\mathbf{I}_t\in \mathbb{R}^{H\times W\times 3}$  by encoder, to update the category-key, representative-key, and discrepancy features in OLDM (see Figure 2(b)).
Update of Category-Key Features Given the feature map  $\mathbf{F}_s$  of the source image  $\mathbf{I}_s$ , we update the category-key features  $\{\mathbf{A}_l \in \mathbb{R}^C \mid l = 1, \dots, L\}$ . We employ decoder to predict a category for every object feature in the map  $\mathbf{F}_s$ . The source image  $\mathbf{I}_s$  has the ground-truth annotation  $\mathbf{Y}_s \in \mathbb{R}^{H \times W \times L}$ . The decoder predicts the category scores  $\mathbf{R}_s(x,y) \in \mathbb{R}^L$ , where  $\mathbf{R}_s(x,y,l) \in \mathbb{R}$  is the score for predicting the object located at  $(x,y)$  of the source image to be the  $l^{th}$  category. We assume the category  $l$  leads to the highest score  $\mathbf{R}_s(x,y,l)$ . The predicted label is compared to the ground-truth label  $\mathbf{Y}_s(x,y)$ . The correct prediction means the object feature  $\mathbf{F}_s(x,y)$  is reliable for updating the category-key feature  $\mathbf{A}_l$  as:
$$
\mathbf {A} _ {l} \leftarrow \mathbf {A} _ {l} + \lambda \mathbf {F} _ {s} (x, y),
$$
$$
s. t. \mathbf {R} _ {s} (x, y, l) = \max  \left\{\mathbf {R} _ {s} (x, y, 1), \dots , \mathbf {R} _ {s} (x, y, L) \right\}, \quad l = \mathbf {Y} _ {s} (x, y), \tag {1}
$$
where  $\leftarrow$  means the update by overwriting.  $\lambda$  is a ratio for controlling the information of  $\mathbf{F}_s(x,y)$  injected into  $\mathbf{A}_l$ . In Eq.(1),  $\mathbf{A}_l$  aggregates the information of an array of object features, which consistently point to the  $l^{th}$  category. It helps  $\mathbf{A}_l$  to capture the representative object style of the  $l^{th}$  category in source domain, which is used to compute discrepancy features.
Update of Representative-Key and Discrepancy Features We use the feature map  $\mathbf{F}_t$  of the target image  $\mathbf{I}_t$  to update the representative-key features in OLDM. For the object feature  $\mathbf{F}_t(x,y) \in \mathbb{R}^C$ , we use the intermediate segmentation head to regress the category scores in  $\mathbf{R}_t(x,y) \in \mathbb{R}^L$ , where intermediate segmentation head takes input as the target feature for regressing the category score map, and  $\mathbf{R}_t(x,y,l) \in \mathbb{R}$  is the score for predicting the object located at  $(x,y)$  of the target image to be the  $l^{th}$  category. We use  $\mathbf{F}_t(x,y)$  to update the representative-key feature  $\mathbf{N}_{l,m}$  as:
$$
\mathbf {N} _ {l, m} \leftarrow \mathbf {N} _ {l, m} + \mathbf {w} _ {l, m} \mathbf {F} _ {t} (x, y), \quad \mathbf {D} _ {l, m} \leftarrow \mathbf {D} _ {l, m} + \mathbf {w} _ {l, m} (\mathbf {A} _ {l} - \mathbf {F} _ {t} (x, y)),
$$
$$
s. t. \mathbf {R} _ {t} (x, y, l) = \max  \left\{\mathbf {R} _ {t} (x, y, 1), \dots , \mathbf {R} _ {t} (x, y, L) \right\} > \gamma ; \mathbf {w} _ {l, m} = \frac {\mathbf {N} _ {l , m} \cdot \mathbf {F} _ {t} (x , y)}{\sqrt {C}}. \tag {2}
$$
The category  $l$  has the highest score in  $\{\mathbf{R}_t(x,y,1),\dots,\mathbf{R}_t(x,y,L)\}$ . We classify the object feature  $\mathbf{F}_t(x,y)$  to the  $l^{th}$  category, thus updating the representative-key features  $\{\mathbf{N}_{l,m}\mid m = 1,\dots,M\}$
in the set  $\mathbb{M}_l$  for the same category.  $\mathbf{w}_{l,m}\in \mathbb{R}$  is a weight, which is calculated as the dot-product between the representative-key feature  $\mathbf{N}_{l,m}$  and the object feature  $\mathbf{F}_t(x,y)$ , whose similarity is measured. A high weight lets object feature contribute more information to representative-key feature.
Unlike decoder supervised by the ground-truth annotations of source images, the intermediate segmentation head is supervised by the pseudo annotations of target images. It may misclassify object feature, thus misleading the update of representative-key features. To remedy this issue, we set a score threshold  $\gamma$  in Eq. (2), to select the reliable object feature  $\mathbf{F}_t(x,y)$ , whose highest score  $\mathbf{R}_t(x,y,l)$  is greater than  $\gamma$ . This threshold only allows the object feature, whose category scores are confident, to update representative-key features. Each representative-key feature aggregates the features of various object instances with similar styles.
In Eq (2), we also update the discrepancy features  $\{\mathbf{D}_{l,m} \in \mathbb{R}^C \mid m = 1, \dots, M\}$ . Here, we subtract the object feature  $\mathbf{F}_t(x,y)$  from the category-key feature  $\mathbf{A}_l$ . Note that  $\mathbf{F}_t(x,y)$  and  $\mathbf{A}_l$  are extracted from the target and source images, respectively. Thus, the subtraction of these features captures the difference between the object styles of in target and source images, which is weighted by  $\mathbf{w}_{l,m}$  to add to the discrepancy feature  $\mathbf{D}_{l,m} \in \mathbb{R}^C$ .
We remark that the update of OLDM is disabled during network testing. The style compensation uses the most updated category-key, representative-key, and discrepancy features in OLDM.
4.2 Style Compensation
The style compensation works during network training and testing. It harnesses the category- and representative-key features to find the appropriate discrepancy features in OLDM, which compensate for the object features of target image (see Figure 2(c)).
Compensation of Object Features Given the object feature  $\mathbf{F}_t(x,y)$  as a query, which is extracted from the target image  $\mathbf{I}_t$ , we use the intermediate segmentation head to predict the category scores  $\mathbf{R}_t(x,y)$ . Next, we select the representative-key and discrepancy features from  $\{\mathbb{M}_1,\dots,\mathbb{M}_K\}$ , where  $\mathbb{M}_k = \{(\mathbf{N}_{k,m},\mathbf{D}_{k,m})\mid \mathbf{N}_{k,m},\mathbf{D}_{k,m}\in \mathbb{R}^C,m = 1,\dots,M\}$  to compensate the object feature  $\mathbf{F}_t(x,y)$ , yielding the compensated feature  $\widetilde{\mathbf{F}}_t(x,y)\in \mathbb{R}^C$  as:
$$
\widetilde {\mathbf {F}} _ {t} (x, y) = \mathbf {F} _ {t} (x, y) + \sum_ {k} \sum_ {m} \mathbf {w} _ {k, m} \mathbf {D} _ {k, m},
$$
$$
s. t. \mathbf {R} _ {t} (x, y, k) \in \max  _ {K} \left\{\mathbf {R} _ {t} (x, y, 1), \dots , \mathbf {R} _ {t} (x, y, L) \right\}; \mathbf {w} _ {k, m} = \frac {\mathbf {N} _ {k , m} \cdot \mathbf {F} _ {t} (x , y)}{\sqrt {C}}, \tag {3}
$$
where  $\max_K$  means to select the top-K relevant categories. We use Eq. (3) to achieve the compensated feature  $\widetilde{\mathbf{F}}_t\in \mathbb{R}^{H\times W\times C}$ , which is used for computing the pseudo annotation for the target image  $\mathbf{I}_t$ .
Computation of Pseudo Annotations Based on the compensated feature  $\widetilde{\mathbf{F}}_t$ , the decoder of the segmentation network predicts the category score map  $\widetilde{\mathbf{R}}_t \in \mathbb{R}^{H \times W \times L}$ . We use the category score map  $\widetilde{\mathbf{R}}_t$  to produce the pseudo annotation  $\mathbf{Y}_t \in \mathbb{R}^{H \times W \times L}$ , where the pixel-wise annotation  $\mathbf{Y}_t(x, y) \in \mathbb{R}^L$  for the pixel located at  $(x, y)$  is determined as: