SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Accuracy (%) on PACS, Office-Home, Office-31, and Digits datasets.
Regime	Method	PACS	Office-Home	Office-31	Digits
acck	accu	hs	acck	accu	hs	acck	accu	hs	acck	accu	hs
OSDA	OSBP [49]	40.6	49.5	44.6	47.1	66.9	55.3	75.8	84.3	77.7	35.6	70.6	40.5
ROS [5]	35.6	66.4	46.4	50.8	77.5	60.8	71.7	80.0	75.6	20.1	48.6	34.9
OD	MSP [20]	38.9	62.5	46.4	52.7	75.6	62.0	49.7	89.2	63.8	17.2	87.1	28.8
LogitNorm [71]	35.1	47.6	38.3	56.3	56.5	56.1	41.0	71.2	52.1	26.8	51.2	35.2
DICE [56]	44.0	53.4	49.2	61.5	58.8	59.9	72.8	61.1	66.4	35.0	47.6	40.3
SFDA	SHOT [35]	51.2	34.9	40.8	52.5	32.4	44.3	84.8	60.2	70.4	27.4	20.3	23.3
AaD [76]	45.1	40.0	42.0	59.4	58.7	58.9	70.1	85.3	76.9	25.6	26.9	26.2
TTA	TTT [58]	36.9	44.6	38.9	52.0	45.9	47.2	35.4	79.6	49.0	40.1	41.1	40.6
Tent [67]	25.2	43.1	31.7	33.6	45.9	38.7	56.0	85.1	67.5	27.2	41.1	32.7
MEMO [80]	37.9	52.3	44.5	49.0	55.6	52.1	59.8	72.7	65.6	21.7	56.1	31.3
OSDG	ADA [66]	54.2	30.9	36.4	67.9	25.4	36.2	85.6	25.2	38.7	57.2	15.1	20.1
ADA+CM [90]	56.4	45.6	43.0	65.0	40.4	48.5	83.0	34.5	48.5	49.2	52.1	39.9
MEADA [81]	54.1	31.4	36.2	67.6	25.7	36.4	85.8	25.1	38.6	57.6	29.8	30.4
MEADA+CM [90]	54.3	46.6	42.7	64.9	40.5	49.6	82.8	41.1	54.7	52.3	46.1	38.7
One Ring-S [77]	43.7	49.4	41.5	56.9	69.0	62.3	67.3	77.0	71.3	33.2	51.3	40.3
OTDG	ERM [60]	52.3	27.0	36.1	66.9	23.7	34.3	85.1	27.0	40.7	56.4	13.0	18.0
CODA (ours)	54.3	63.8	58.3	59.7	74.6	66.3	87.5	75.4	81.0	31.5	60.1	41.3
(a)ERM[60]
Figure 3: Performance comparisons of different methods as testing proceeds on the PACS dataset.
(b) MSP [20]
(c) TAST [24]
(d) CODA (ours)
9.7% for Office-31, and 1.0% for Digits. If we focus on the hard generalization tasks, such as PACS, CODA exhibits larger performance gains than on other tasks. Moreover, three trends can be observed: (1) Compared to OSDA and SFDA methods that usually optimize with target data offline, CODA achieves superior performance in an online adaptation manner. (2) The  $acc_k$  and  $acc_u$  of Tent [67] and LogitNorm [71] exhibit significant imbalance as both methods tend to predict all data as known-class samples (i.e. shortcut learning). This verifies the benefits of CODA in mitigating shortcut learning. (3) OD methods achieve very competitive results compared to other types of baselines. The rationale is that they usually do not involve test-time adjustment and thus have better stability. (4) The performance of different types of baseline methods varies across benchmarks. For instance, MEMO (TTA) achieves the second-best result in PACS but has relatively inferior performance in Office-Home and Digits. Instead, CODA exhibits consistent improvements on all benchmarks.
In addition, we provide performance comparisons of different methods (i.e., ERM [60], MSP [20], TAST [24], and our CODA) as testing proceeds on the PACS dataset (trained on domain Art Painting). To facilitate a fair comparison, both MSP and TAST will apply to the models that have been trained by our source compaction stage. Figure 3 shows that ERM and TAST substantially increase  $acc_{k}$  and maintain it at a very high level, which severely impedes the improvements of  $hs$ . Interestingly, as the number of testing epochs increases, TAST underperforms
compared to ERM. By contrast, CODA dynamically harmonizes the relations between  $acc_{k}$  and  $acc_{u}$  (i.e., suppresses  $acc_{k}$  and hence allows  $acc_{u}$  to grow), which is reflected by the monotonic increase of  $hs$ . Figure 4 shows Grad-CAM [52] visualizations of baseline (ERM) and our method (CODA) on the PACS dataset. We can see that the hot zones activated by CODA are more complete and reasonable, providing a reliable semantic understanding of the foreground object.
Figure 4:Top: ERM. Bottom: CODA.
Figure 5: Predictions from models trained with and without target disambiguation.
Ablations of key components in CODA. We carry out ablation studies in Table 3, evaluating the effect of source compaction (SC) and target disambiguation (TD) proposed in CODA. When we exclude SC from CODA, model predictions are made based on the entropy of the output in conjunction with a predetermined threshold [90]. From the table, we can observe that adding SC and TD could improve the generalization per
Table 3: Ablation of CODA on four classification benchmarks.  $hs(\%)$  is reported.
SC	TD	PACS	Office-Home	Office-31	Digits
×	×	36.1	34.3	40.7	18.0
✓	×	51.2	64.8	75.3	39.0
×	✓	49.8	61.7	72.3	37.5
✓	✓	58.3	66.3	81.0	41.3
formance, which verifies their individual contributions for solving domain shifts and open classes. Moreover, our method that integrates both SC and TD achieves the best performance, revealing the synergistic effect between the two components.
Analysis on target disambiguation. In Figure 5, we plot the predictions on known- and unknown-class test samples from the models trained with and without target disambiguation. The left image is a battery (known), and the right image is a clipboard (unknown). For the known classes, a model lacking disambiguation often makes uncertain predictions, especially for hard samples that bear high resemblance to other classes. For the unknown class, a model without disambiguation tends to give a high response to an incorrect class and suppress responses to other classes. In contrast, our target disambiguation stage can achieve more accurate predictions by recovering the semantic relationships among classes from unlabeled data, thereby enhancing the model's generalization performance under both domain shift and open classes. In Figure 6 (a)-(b), we investigate the the impact of varying the test batch size on three methods, TAST [24], FAU [69], and CODA (ours). As the batch size varies, CODA consistently delivers superior performance compared to TAST and FAU, revealing the advantages of the proposed online adaptation strategy.
Analysis on unknown-aware training objective. We empirically compare different unknown-aware training objectives discussed in Section 2 and Section 3.1, i.e., Eq.(1)-(4) and their combinations. For a fair comparison, we do not involve any TTA strategies including our target disambiguation. The results are reported in Table 4. We can observe
Table 4: Analysis on unknown-aware training objective.
Method	PACS	Office-Home	Office-31	Digits
Eq. 1	36.1	34.3	40.7	18.0
Eq. 2	41.1	58.9	63.0	39.3
Eq. 3	46.8	62.4	72.6	38.6
Eq. 4	43.2	60.3	70.7	37.4
Eq. 3 + Eq. 4	51.2	64.8	75.3	39.0
that our full source compaction is clearly better than its variants, revealing the superiority of our optimization procedures on both real known-class samples and virtual unknown-class samples.
Analysis on unknown classes. In Figure 6 (c)-(d), we study the impact of varying the number of known classes on three methods, ERM [60], One Ring-S [77], and CODA (ours). Note that the total number of classes (i.e.  $|\mathcal{C}_s \cup \mathcal{C}_t|$ ) remains unchanged. Even when the number of known classes is small, CODA still exhibits superior performance. This advantage remains consistent as the number of known classes changes. Consequently, CODA is capable of handling extreme scenarios.
Feature visualization. We use  $t$ -SNE [59] to visualize the features of four models on Office-31 dataset, i.e., ERM, One Ring-S, Source Compaction, and CODA (full). The results are depicted in Figure 7, where various colors, excluding gray, signify different known classes, and gray points represent all unknown classes. It is noteworthy that the embedding features learned by two baselines (ERM and One Ring-S) fail to present a clear separation, resulting in ambiguous boundaries among
Figure 6: (a)-(b) The influence of test batch size. (c)-(d) The influence of varying the number of known classes. Figures (a) and (c) are plotted based on the Office-Home dataset, while figures (b) and (d) are derived from the PACS dataset.
(a) ERM
Figure 7: t-SNE visualization of the learned features on the Office-31 dataset.
(b) One Ring-S [77]
(c) Source Compaction
(d) CODA (full)
different classes particularly between known and unknown ones. Instead, CODA can learn the intrinsic structure ("manifold") of the target samples, providing more discriminable clustering patterns.
Domain Generalization (DG). DG is concerned with the training of a model using data from either multiple or a single source domain, with the goal of achieving good generalization to previously unseen target domains. Mainstream approaches usually involve invariant learning and robust learning with elaborate training objectives. Based on the focus of this paper, we classify existing methods into three categories and provide descriptions as follows. (1) Closed-set DG. Existing methods can be roughly divided into four categories: feature matching-based [32, 31, 40, 91, 83], decomposition-based [46, 45, 13, 39, 54, 36, 78], augmentation-based [65, 85, 86, 74, 41, 88, 8], and meta-learning-based approaches [29, 33, 30, 37, 7]. (2) Open-set DG. It is worth noting that very few works have delved into the exploration of DG in open-set scenarios. A handful of recent studies [53, 90, 77, 9] started to consider the existence of both known and unknown classes in novel DG settings, such as Open-Set DG (OSDG) [90, 77]. For example, Yang et al. [77] hold the view that any category other than the ground-truth can be considered as part of the unknown classes. Chen et al. [9] first activate the unknown's logit via an unknown-aware training loss and then introduce a test-time adjustment strategy to refine the model prediction. Zhu et al. [90] rely on heuristic thresholding mechanisms to distinguish known- and unknown-class samples. (3) DG by test-time adaptation. According to Dubey et al. [14], a model trained solely on source data will inevitably experience an "adaptivity gap" when it is directly employed on target domains, emphasizing the necessity of on-target adaptation. Grounded on this insight, several recent works [79, 73, 72, 11] resort to TTA for mitigating the adaptivity gap, such as adaptive risk minimization [79], energy-based sample adaptation [72], and improved self-supervised learning tasks [11].
Test-Time Adaptation (TTA). TTA [34] is an emerging paradigm that has demonstrated immense potential in adapting pre-trained models to unlabeled data during the testing phase. A plethora of approaches [35, 58, 67, 23, 43, 10, 80, 69] have been developed to improve the predictions of source-trained models on target domain with online training/adaptation strategies. TTT [58] introduces self-supervised learning tasks (e.g. rotation classification) to both source and target domains. Tent [67] leverages the batch statistics of the target domain and optimizes the channel-wise affine parameters by entropy minimization. T3A [23] proposes to use class prototypes for adjusting predictions and introduces a support set to memorize representative and reliable samples. TAST [24] improves T3A by proposing a nearest neighbor information induced self-training framework.
Out-of-Distribution Detection (OD). OD [75], which seeks to identify novel instances that the model has not encountered during training, is close to OTDG setting. Prevailing OD methods center on creating OOD scoring functions, for example, confidence-based techniques [4, 20, 21], distance-based scores [27, 51, 57], and energy-based scores [38, 55]. Although promising, the OD approach is limited to binary classification problems and lacks the capability to effectively explore domain shift and open class challenges in the test data.
There are also other topics related to open-world machine learning [44] that bear certain relevance to our work, including open-set recognition [50, 62], novel class discovery [15, 61], zero-shot learning [47, 70], and class-incremental learning [6, 82], to name a few. Compared to previous methods, our work addresses two types of open-world situations (i.e., domain shift and open class), supporting generalization capabilities consistently throughout the training and inference phases.
In this paper, we solve the problem of OTDG where both domain shift and open classes may concurrently arise on the unseen test data. We introduce a two-stage framework (CODA) for efficiently learning what we don't know in the wild. At the training stage, we compact the embedding of source known classes and thus reserve space for target unknown classes. In the testing phase, we introduce a training objective to mitigate the optimality gap between domains while avoiding catastrophic forgetting. Empirically, CODA achieves superior performance on standard DG benchmarks.
This work was partially supported by Hong Kong Research Grants Council under Collaborative Research Fund (Project No. HKU C7004-22G).
[1] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In ICML, pages 145-155, 2020.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893, 2019.
[3] Shai Ben-David, Tyler Lu, Teresa Luu, and David Pál. Impossibility theorems for domain adaptation. In International Conference on Artificial Intelligence and Statistics, pages 129-136, 2010.
[4] Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, pages 1563-1572, 2016.
[5] Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In ECCV, 2020.
[6] Francisco M Castro, Manuel J Marín-Jiménez, Nicolas Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV, 2018.
[7] Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaqing Liu, and Yizhou Yu. Compound domain generalization via meta-knowledge encoding. In CVPR, pages 7119-7129, 2022.
[8] Chaoqi Chen, Luyao Tang, Feng Liu, Gangming Zhao, Yue Huang, and Yizhou Yu. Mix and reason: Reasoning over semantic topology with data mixing for domain generalization. NeurIPS, 35:33302-33315, 2022.
[9] Chaoqi Chen, Luyao Tang, Leitian Tao, Hong-Yu Zhou, Yue Huang, Xiaoguang Han, and Yizhou Yu. Activate and reject: Towards safe domain generalization under category shift. In ICCV, pages 11552-11563, 2023.
[10] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, pages 295-305, 2022.
[11] Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In CVPR, 2023.
[12] Noam Chomsky. Aspects of the Theory of Syntax, volume 11. MIT press, 2014.
[82] Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In CVPR, 2022.
[83] Hong-Yu Zhou, Yizhou Yu, Chengdi Wang, Shu Zhang, Yuanxu Gao, Jia Pan, Jun Shao, Guangming Lu, Kang Zhang, and Weimin Li. A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nature Biomedical Engineering, 7:743-755, 2023.
[84] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. TPAMI, 2022.
[85] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In AAAI, pages 13025-13032, 2020.
[86] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV, pages 561-578, 2020.
[87] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. TIP, 30:8008-8018, 2021.
[88] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021.
[89] Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In ICML, pages 27222-27244, 2022.
[90] Ronghang Zhu and Sheng Li. Crossmatch: Cross-classifier consistency regularization for open-set single domain generalization. In ICLR, 2022.
[91] Wei Zhu, Le Lu, Jing Xiao, Mei Han, Jiebo Luo, and Adam P Harrison. Localized adversarial domain generalization. In CVPR, pages 7108-7118, 2022.