CODA: Generalizing to Open and Unseen Domains with Compaction and Disambiguation
Authors: Chaoqi Chen
Luyao Tang
Yue Huang
Xiaoguang Han
Yizhou Yu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Domain generalization
domain shift
open class
source compaction
target disambiguation
TL;DR: We propose a principled framework (CODA) for a new and challenging domain generalization setting where both domain shift and open class occur on test data.

Abstract: The generalization capability of machine learning systems degenerates notably when the test distribution drifts from the training distribution. Recently, Domain Generalization (DG) has been gaining momentum in enabling machine learning models to generalize to unseen domains. However, most DG methods assume that training and test data share an identical label space, ignoring the potential unseen categories in many real-world applications. In this paper, we delve into a more general but difficult problem termed Open Test-Time DG (OTDG), where both domain shift and open class may occur on the unseen test data. We propose Compaction and Disambiguation (CODA), a novel two-stage framework for learning compact representations and adapting to open classes in the wild. To meaningfully regularize the model's decision boundary, CODA introduces virtual unknown classes and optimizes a new training objective to insert unknowns into the latent space by compacting the embedding space of source known classes. To adapt target samples to the source model, we then disambiguate the decision boundaries between known and unknown classes with a test-time training objective, mitigating the adaptivity gap and catastrophic forgetting challenges. Experiments reveal that CODA can significantly outperform the previous best method on standard DG datasets and harmonize the classification accuracy between known and unknown classes.

CODA: Generalizing to Open and Unseen Domains with Compaction and Disambiguation
Chaoqi Chen 1-
Luyao Tang²* Yue Huang?
Xiaoguang Han Yizhou Yu!
The University of Hong Kong Xiaaeen University
The Chinese University of Hong Kong (Shenzhen)
egchen19940gaa11.com lytang@stu... edu en, yhuang2010@xmu edu.cc.edducc banziaoguangécuhk. edu. cn, yizhouy org
Abstract
The generalization capability of machine learning systems degenerates notably when the test distribution drifts from the training distribution. Recently, Domain Generalization (DG) has been gaining momentum in enabling machine learning models to generalize unseen domains. However, most DG methods assume that training and test data share an identical label space, ignoring the potential unseen categories in many real-world applications. In this paper, we delve into a more gen- eral but difficult problem termed Open Test-Time DG (OTDG), where both domain shift and open class may occur on the unseen test data. We propose Compaction and Disambiguation (CODA), novel two-stage framework for learning compact representations and adapting to open classes in the wild. To meaningfully regu- larize the model's decision boundary, CODA introduces virtual unknown classes and optimizes new training objective to insert unknowns into the latent space by compacting the embedding space of source known classes. To adapt target samples to the source model, we then disambiguate the decision boundaries between known and unknown classes with test-time training objective, mitigating the adaptivity gap and catastrophic forgetting challenges. Experiments reveal that CODA can significantly outperform the previous best method standard DG datasets and harmonize the classification accuracy between known and unknown classes.
Introduction
The ability to generalize to unseen environments is considered a key signature of human intelli- gence [12]. While deep neural networks have achieved great success in many machine learning problems, they are brittle to distribution shifts between training and test domains, which often occur real-world applications. For example, when deploying object recognition systems autonomous vehicles, the ever-changing weather conditions (e.g. fog, rain, and snow) may deteriorate the per- formance and raise concerns about their reliability over time. This motivates challenging scenario named Domain Generalization (DG) [68, 84], which extrapolates learning machines related yet previously unseen test domains by identifying the common factors from available source data.
From the perspective of representation learning, the mainstream paradigm for DG includes invariant risk minimization [2, 1, 89]. domain alignment [31], feature disentanglement [45, 36, 78], meta- learning [29, 30], and augmentation-baan based invariant prediction [66, 74, 88]. In spite of the significant progress in DG, the adaptivity gap [14] between source and target domains naturally exists and emerges as inevitable challenge. Therefore, some prior efforts [23, 24, 72, 11] strive to adapt
First authors contributed equally. Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the source- source-trained model through the lens of temt-time adaptation [34], which uses unlabeled target samples in an online manner. While these methods alleviate the adaptivity gap to some extent, there is still no guarantee for any specific domain, especially if the domain divergence is large [3]. Moreover, most existing DG methods are designed with the assumption that the label space in the source and target domain are identical, which is too restrictive to be satisfied in practice.
To enable machine learning systems to be resilient an open world, we aim at more practical yet under-explored problem named Open Test-time Domain Generalization (OTDG), where both domain shift and open class occur in the unseen target data. The primary challenge of the proposed problem lies in addressing two critical aspects: (1) model generalization with an incomplete training label space, and (2) online model adaptation with asymmetric test label space.
Motivated by this, we propose a novel framework for OTDG, termed Compaction and Disambigua- tion (CODA). Our key idea is to enforce constraints the decision boundaries during training using labeled source data and refine them during test time using unlabeled target data. To regularize the model's decision boundary and make the model expandable, CODA introduces set of virtual un- known classes and optimizes a novel training objective in conjunction with the standard cross-entropy loss. The response to known- and unknown-class logits will be activated for both real and synthesized samples. This process embeds unknowns into the latent space by compacting the embedding space of known source classes, thereby reserving sufficient latent space for target unknown classes. To disambiguate the decision boundaries between known and unknown classes, we propose novel prototype-basee test-time adaptation pipeline. Specifically, the test-time classification should be subjected to three constraints: (1) consistency between the predicted class distributions and the estimated class conditionals, (2) class-wise sample reliability for ensuring the quality of target pseudo labels, and (3) semantic consistency between source-trained and present model predictions.
The main contributions are summarized as follows:
We propose CODA, simple and effective DG framework to mitigate domain shift and identify open classes in unseen test environments.
We introduce a virtual unknown optimization process to make the model expandable for open classes, and a test-time training objective to match the real test data to corresponding known and unknown class patterns.
We conduct extensive experiments and demonstrate that CODA outperforms previous methods on a series of DG benchmarks.
2 Preliminaries
Problem setup. Let us formally define the OTDG problem. We have access to a source domain D. [(x*, // of n, labeled data points and multiple unseen target domains D. = {xx))== of ne unlabeled data points. Let C, and C, be the source and target class sets, respectively. In OTDG, we have C, C C, and CT = C. \ C, is referred to as unknown classes. Assume that X' is the input space, Z is the latent space, and ) is the output space. The predictor f h o g is comprised of a featurizer that makes predictions based on the extracted features. The goal of OTDG is to find a predictor f y that generalizes well to all unseen target domains. Although the labeling function hy is unknown, we assume that we have access to unlaheled instances from D, at test time. In addition, we present the comparison of different problem settings in Table 1.
X' + Z that learns to extract embedding features, and a classifier h 2 + y
Unknown-aware training. To solve OTDG, simple baseline is to train a (C.|+ID-way classifier [90, 77, 9], where the additional dimension is introduced to identify the unknown. Formally, we define the standard cross-entropy loss as:
exp(fx(x))
LCE LCE(S(x),g) -log
(1)
Exelc.|+1 (x)(
where /(x E RIC,|+1 denotes the network's logit and fx(() is the k-th element of [xx correspond- ing to the ground-truth label y. As shown in Figure/ I (a), however, such optimization fails to activate the network's response to unknown classes. In this work, we first introduce a simple yet very effective baseline [9] for OTDG, Their key idea is to directly activate the unknown's 's logit by optimizing its

---

Table I: Comparison of related machine learning problems. DA refers to Domain Adaptation and OOD stands for Out-of-Distributt 'One-pass' indicates that target domain data only passes the network once during the whole process including the training and testing phases.
Training
Test-time
Problem Setting
One-pass
Training Data
Training Loss
Testing Loss
Domain shift
Open class.
Adaplivity gap
Open-Set
L(x., Me) X(x
Source: Free DA
Xt
((xxs)
Nax 1/
C(xss M6)
Dest-Time Adaptation
Na. Me
L(x.. Me)
[(x))
Test-Time DG
N.. Mo
L(x, Mo)
((x)
X
Open-
Xa: Me
L(x., Ma)
OIDG (Ours)
C(x.,
((x.)
(a)
(b)
Figure 1: The decision boundaries learned by different unknown-aware training objectives: (a) Eq. 1, (b) Eq. 2, and (c) Eq. 3 Eq. 4. These toy examples are generated by scikit-learn toolkit. Yellow, blue, orange, and pink points represent the known-class samples, while black points are unknown-class samples. Different clusters of back points stand for different unknown classes.
likelihood, without affecting the ground-truth classification. For source sample (x., Us) E Dx,, we minimize the negative log-likeet ligellkl w.r.t. unknown logit increase the unknown probability,
exp(fu(x))
Cuxr(f(x),v) 3) Ccn(f(x), log
Eceje, +1.c exp(fx(x))
where (x) is the unknown's logit. Eq. 2 makes the unknown class probability respond any input sample, irrespective of its class label. Since the learning process is driven by the cross-entropy loss related the ground-truth category, Eq. does not hurt the performance of known classes.
Proposed Method
We propose CODA (Figure 2), a simple two-stage OTDG framework for discovering unknown classes with the help of known ones. The specific implementation of the two stages is described as follows.
3.1 Training-Time Time Source Compaction
The source compaction stage preserves sufficient space for the upcoming unknown classes without using real target data during training. Since we have no a priori knowledge or assumptions about the characteristics (e.g., number and attribute) of unknown classes, is prohibitively difficult to learn meaningful representations beforehand. To begin with, we make mild assumption that the unknown classes should be far away from all known classes in the embedding space (category shift). In practice, there are two choices for accommodating these unknown classes: (1) low-density region (Figure (b)), and (2) netween-class region (Figure 1 (c)). The former is intuitive and could be achieved via Eq. 2, which naturally corresponds to these regions. However, the multi-modal structure of unknown-class data is underspecified, i.e., different unknown classes should not occupy the same region. By contrast, the latter able to implicitly separate different unknowns and model the relationships between known and unknown classes. To make the model expandable, our key idea

---

real known-class
real known-class
Compaction
embedding
Eq. (1)
Eq. (2)
Source
g
h
Logits
Eq. (3)
Eq. (4)
featurizer
clossifier
known
unknown
real known-closs
virtual unknown-class
Prototype-based classifier
Semantic
Target
h
fo
consistency
Source- trained model (fixed)
Self- training
feature
logits
Memory bank
N(x)
Estimated class conditionals
Disambiguation
Nearest neighbor
Figure 2: Overview of the proposed CODA, which consists of novel components: (1) Training- time source compaction to make the model expandable for open classes; (2) Test-time target disambiguation to discriminate the decision boundaries with a test-time training objective.
to compact the source embedding space by (i) making real known samples give response to both known and unknown classes, and (ii) inserting virtual unknown samples between known-class pairs.
Optimization on real known-class samples. addition to the original source classes, we introduce set of virtual classes C, within the source embedding space to mimic the existence of unknown classes. To accommodate the virtual unknown classes, we regularize the embedding space by pushing known-class samples closer to the decision boundaries:
exp(fa(x))
(f CCE(f(x),)) log Enek, +Ce.leev exp fc(())
where (x(x) is the unknown's logit, having the largest activation among C..
Optimization on virtual unknown-class samples. Despite the activation of the network's response to the introduced unknown classes, the "winner-takes-ae nature of softmax-base classification still leaves the unknown category unable to compete effectively with known ones (cf. Figure 2). To mimic test environments, we synthesize virtual unknown samples. Technically, we synthesize unknown samples at the feature level for model regularization, without using external data, i.e., dashed points in Fig. 1 (c). We introduce manifold mixup [64] for synthesizing unknowns in the between-clase regions, which are less confident for current decision boundaries. For two random samples x, and x, from different classes, we mix their embedding features as: 2 μ (xx) + (1 14) g(x;), where the mixing coefficient. The optimization objective for the synthesized unknown is defined as:
exp(h (2))
(himua(h(2,,)))) = Cce(C(())))))) - llog Σ Lee,C,++CC.).of5 (x)(h,(())
where ÿ represents the label of regarding unknown class, i.e., having the largest activation among C, hz)(2) is the known's 's logit, having the largest activation among C, The first term in Eq. 4 is standard self-training loss. Similar to Eq. 3, the second term activates the response of 2 to most related known class. In essence, apart from the standard classification loss (the first term in Eq. 3 and Eq. 4), we activate the response of the real known class towards unknowns (Eq. 3) and the response of the virtual unknown class towards the known ones (Eq. 4).

---

3.2 Test-Time Target Disambiguation
Although we have allocated the embedding space for unknown classes, how deploy the source- trained model on real test data is yet to be thoroughly studied. In particular, we identify two major challenges: (i) optimality gap between source and target domains, and (ii) catastrophic forgetting in open and dynamic test environments. In OTDG, we define the optimality gap as follows.
Definition 1 (Optimality Gap) Let H. S {hlh : Z + y} be the hypothesis class. Es(-) and eT()) denote the expected risk on source and target domains. For any hypothesis h, we have (h(h') < ET(h"), where h = arg minneH és (h) + ET ET(h) and h' = arg minnen (T))..
Definition suggests that it is not feasible to find a universal optimal classifier that applies to both source and target domains. In that sense, the classifier, initially trained on source data, needs additional refinement to adapt effectively to the target patterns. Technically, we resort to TTA [34] to mitigate the above issues using unlabeled test data an online manner. Conventional training-based TTA methods [67] usually need batches of data for self-training (e.g. entropy minimization) and/or heuristic self-supervision tasks [10]. On the other hand, training-free methods [23] require expensive tweaking of the threshold and only bring marginal performance gain. Moreover, these approaches struggle to handle open-class scenarios, making them susceptible negative transfer (i.e. semantic misalignment). To address these challenges, our proposed target disambiguation stage aligns the unlabeled target samples to their corresponding class patterns through the following process.
We construct memory bank S {8), gic.++i..ly for memorizing the embedding z and logits
f() (or h(z)) of target samples. We compute a set of class prototypes {p*} [c.|+ based on logits in S. Following [23, 24), the memory bank is initialized with the weights of the linear classifier. Let
((x) be the Nearest Neighbor (NN) of x in S. For each test sample x', we search its NN in S by: ((x) {z () xx,, z) 0NN). J. where sim(-) ) is the cosine similarity and 6NN is threshold to control the number of NN. The model predictions would be given by the similarity between sample embedding and the class prototype. i.e., p(y|x) sim (p,, 2). Formally, for E (x)), the likelihood of the prototype-based classifier assigning z to the k-th class can he calculated as follows:
exp(-sim(b(z)) (b)))))))/)) ply l/2z= = Σ. (oon-ssmm(())) P.)/T)' k = 1,2, |C.| + [Col:
(5)
where T a temperature scaling parameter. We estimate the class conditionals with WithW(sx (x) as:
P
Σ
1 (arg max = k),
(6)
[(x)
xxN(x)
where 1.) is an indicator function. Then, we can update the global class prototype computed from the whole S moving-average style,
P& μP: (1-1))p:
where E [0,1] is preset scalar and Pa can be regarded as the local class prototype. Given batch of test samples B,, we use self-training for model updating (g and h), i.e., minimizing the cross-entropy loss between classifier's prediction f(x) and the estimated class prior distribution p&:
(x)(x)
Σ CCR(PE, f(x)) FEB
(8)
B.
Semantic consistency. To resist catastrophic forgetting during the online adaptation process, we enforce semantie consistency between the output of fo (source-trained model) and fi (target model) by optimizing the cross-entropy loss between their predictions:
(sc(x) = (ff(x))) log a (f)()))
where / represents the number of iterations and fo is fixed throughout the testing phase.
Reliable sample selection. In the early stage of training, the estimation of target pseudo labels may be unreliable and thus is risky to error accumulation. To improve the quality of pseudo labels and
We omit the subscript č for simplicity.
5

---

reduce the influence of false class estimations, we introduce an entropy- based weighting strategy to select reliable samples. Specifically, we define the scoring function as follows:
S(x) = (x) < 0o) exp(00 H (x)),
(10)
where (.) is the Shannon entropy of sample and dg is pre-defined threshold. In this way, can allocate larger weights to target samples with lower uncertainties and smaller weights to those with higher uncertainties, effectively prioritizing more confident predictions.
Test-time training objective. Formally, the overall optimization objective can be formulated as:
(xTT(x) ((x) + >x
(11)
where is trade-off parameter. The proposed LrTT embraces the complementary strengths of parametric ssftmax-basedd and non-parametric (prototype-baeed) classifiers.
4 Experiments
4.1 Setup
Benchmarks. We conduct extensive experiments on four standard DG benchmarks to verify the effectiveness of CODA. (1) PACS [28] has 9,991 images and presents remarkable distinctions in image styles. It is comprised of four domains each with seven classes, i.e, Photo, Art Painting, Cartoon, and Sketch. Dog, elephant, giraffe, and guitar are used as C, while the remaining classes are C₁ (2) Office-Home [63] is gathered from both office and home environments, and its domain shifts originate from variations in viewpoint and image style. It has 15,500 images of 65 classes from four domains, i.e,, Artistic, Clipart, Product, and Real World. Arranged in alphabetical order, the initial 15 classes are designated as C.. and the remaining 50 classes are assigned to C₁¹ (3) Office-31 [48] encompasses 31 classes harvested from three distinct domains: Amazon, DSLR, and Webcam. The 10 classes shared by Office-31 and Caltech-255 [18] are used as C.. In alphabetical order, the final 11 classes, combined with C., constitute C, (4) Digits, a dataset varying in background, style, and color, encompasses four domains of handwritten digits, including [2]ST[26], MNIST- SVHN(42), USPS[22), and SYN [17]. We utilize MNIST as the source domain, while the other datasets serve as target domains. Numbers from 0 to make up C..
Evaluation Protocols. In line with prior works [5, 90, 77]. we utilize the H-score (hs) [16] as our main evaluation criterion. The hs score balances the significance between known and unknown classes, emphasizing that both groups should have high and equivalent accuracy. Additionally, the hs score circumvents the trivial solution where a model would predict all samples as known classes. The classification accuracy for both known (accy) and unknown (acc,) classes are also reported.
Implementation Details. For PACS, Office- Home, and Office-31, we employ ResNet-18 [19], pre-trained on ImageNet, as the backbone network. For Digits, we employ the LeNet [25] with the architecture arranged as
pool-con--pool fe-fffsff.f..aa.. The training performed using SGD with momentum of 0.9 for 100 epochs, and we set the batch size to 64. Our experiments are built upon Dassl [87] (a PyTorch toolbox developed for DG), covering aspects of data preparation, model training, and model selection. We report the means over 5 runs with different random seeds.
4.2 Baselines
In experiments, we empirically compare CODA against five types of baselines: (1) OSDG: CM [90]
and One Ring-S [77]. (2) OSDA: OSBP [49] and ROS [5]. (3) OD: MSP [20], LogitNorm [71],
and DICE [56]. (4) SFDA: SHOT [35] and AaD [76]. (5) TTA: TTT [58], Tent [67], MEMO [80],
TAST [24], and FAU [69]. Since TTA methods are incapable of directly handling unknown-class
samples, we adopt the approach from [90], using the entropy of the softmax output as the final
prediction. For ERM [60], we follow the same strategy for identifying unknowns.
4.3 Results
Our results are summarized in Table 2. For each dataset, CODA outperforms all compared methods by a considerable margin in terms hs. For instance, in comparison to the previous hest-performing OSDG baseline [77], CODA achieves increases in hs by 16.8% for PACS, 4.0% for Office- Home,

---

Table 2: Accuracy (%) PACS, Home, Office-31, and Digits datasets.
PACS
Office-Home
Office-31
Digits
Regime
Method
aCCk:
aCC
h8
aCC;
BCCs
h8
a0ck
acC,
has
aOCk
acc,
hs
OSBP [49]
40.6
49.5
44.6
47.1
66.9
55.3
75.8
84.3
77.7
35.6
70.6
40.5
OSDA
ROS [5]
35.6
66.4
46.4
50.8
77.5
60.8
71.7
80.0
75.6
20.1
48.6
34.9
MSP [20]
38.9
62.5
46.4
52.7
75.6
62.0
49.7
89.2
63.8
17.2
87.1
28.8
LogitNorm [71]
35.1
47.6
38.3
56.5
56.1
41.0
71.2
52.1
26.8
51.2
35.2
DICE [56]
44.0
53.4
49.2
61.5
58.8
59.9
72.8
61.1
66.4
35.0
47.6
40.3
SHOT [35]
51.2
34.9
40.8
52.5
32.4
44.3
84.8
60.2
70.4
27.4
20.3
23.3
SFDA
AaD [76]
45.1
40.0
42.0
59.4
58.7
58.9
70.1
85.3
76.9
25.6
26.9
26.2
TTT [58]
36.9
44.6
38.9
52.0
45,9
47.2
79.6
49.0
41.1
40.6
TTA
Tent [67]
25.2
43.1
31.7
33.6
45.9
38.7
56.0
85.1
67.5
27.2
41.1
32.7
MEMO [80]
37.9
52.3
44.5
55.6
52.1
59.8
72.7
65.6
21.7
56.1
31.3
ADA [66]
54.2
30.9
36.4
67.9
25.4
36.2
85.6
25.2
38.7
57.2
15.1
20.1
ADA+CM [90]
56.4
45.6
43.0
65.0
40.4
48.5
83.0
34.5
48.5
49.2
52.1
39.9
MEADA [81]
54.1
31.4
36.2
67.6
25.7
36.4
85.8
25.1
38.6
57.6
29.8
30.4
OSDG
MEADA+CM [90]
54.3
46.6
42.7
64.9
40.5
49.6
82.8
41.1
54.7
52.3
46.1
38.7
One Ring-S [77]
43.7
49.4
41.5
56.9
69.0
62.3
67.3
77.0
71.3
33.2
51.3
40.3
ERM [60]
52.3
27.0
36.1
66.9
23.7
34.3
85.1
27.0
40.7
56.4
13.0
18.0
OTDG
CODA (ours)
54.3
63.8
58.3
59.7
74,6
66.3
87.5
75.4
81.0
31.5
60.1
41.3
Accuney
ACcunaser
1400
Mome
Epoch
Epoch
20 10
(a) ERM [60]
(b) MSP [20]
(c) TAST [24]
(d) CODA (ours)
Figure 3: Performance comparisons of different methods as testing proceeds on the PACS dataset.
9.7% for Office-11, and 1.0% for Digits. If focus the hard generalization tasks, such as PACS, CODA exhibits larger performance gains than on other tasks. Moreover, three trends can be observed: (1) Compared to OSDA and SFDA methods that usually optimize with target data offline, CODA achieves superior performance in an online adaptation manner. (2) The acc; and acca of Tent [67] and LogitNorm [71] exhibit significant imbalance both methods tend predict all data as known-class samples (i.e. shortcut learning). This verifies the benefits of CODA in mitigating shortcut learning. (3) OD methods achieve very competitive results compared other types of baselines.
The rationale is that they usually do not involve test-time adjustment and thus have better stability. (4) The performance of different types of baseline methods varies across benchmarks. For instance, MEMO (TTA) achieves the second-best result in PACS but has relatively inferior performance in Office-Home and Digits. Instead, CODA exhibits consistent improvements on all benchmarks.
In addition, we provide performance comparisons of dif- ferent methods (i.e., ERM [60], MSP [20], TAST [24], and our CODA) as testing proceeds on the PACS dataset (trained domain Ari Painting). To facilitate fair com- parison, both MSP and TAST will apply to the models that have been trained by our source compaction stage, Figure 3 shows that ERM and TAST substantially increase acc; and maintain it at a very high level, which severely impedes the improvements of hs. Interestingly, the number of testing epochs increases, TAST underperforms compared to ERM. By contrast, CODA dynamically harmonizes the relations between acc; and acc, (i.e., suppresses ace; and hence allows acc,, to grow), which reflected by monotonic increase of hs. Figure 4 shows Grad-CAM [52] visualizations of baseline (ERM) and our method (CODA) on the PACS dataset. We can see that the hot zones activated by CODA are more complete and reasonable, providing reliable semantic understanding of the foreground object.
Figure 4: Top: ERM. Bottom: CODA.
7

---

w/o Disambiguation
w/ Disambiguation
wio Disambiguation
w/ Disambiguation
Battery
Backgack
Unknown
Calculator
Unknown
Backpack
Battery
Gipboard
Ciphoard
Glipboard
Clipboerd
Unknown
Calendar
Computer
Uninown
Compuler
Computer
Buckel
Figure 5: Predictions from models trained with and without target disambiguation.
4.4 Ablation Studies
Ablations of key components in CODA. We earry out ablation studies in Table 3, evaluating the effect of source compaction (SC) and target disambiguation (TD) proposed in CODA. When we exclude SC from CODA, model predictions are made based on entropy the output in conjunction with predetermined threshold [90]. From the table, we can observe that adding SC and TD could improve the generalization per- formance, which verifies their individual contributions for solving domain shifts and open classes. Moreover, method that integrates both SC and TD achieves the best performance, revealing the synergistic effect between the two components.
Table 3: Ablation of CODA on four classification benchmarks. hs (%) reported.
SC TD
PACS
Office-Home Office-31
Digits
X
x
36.1
34.3
40.7
18.0
V
X
51.2
64.8
75.3
39.0
X
V
49.8
61.7
72.3
37.5
V
v
58.3
66.3
81.0
41.3
Analysis target disambiguation. In Figure 5, we plot the predictions on known- and unknown- class test samples from the models trained with and without target disambiguation. The left image is a battery (known), and the right image is a cl clipboard (unknown). For the known classes, a model lacking disambiguation often makes uncertain predictions, especially for hard samples that bear high resemblance other classes. For the unknown class, a model without disambiguation tends to give high response to an incorrect class and suppress responses to other classes. contrast, our target disambiguation stage achieve more accurate predictions by recovering the semantic relationships among classes from unlabeled data, thereby enhancing the model's generalization performance under both domain shift and open classes. In Figure 6 (a)-(b), investigate the the impact of varying the test batch size on three methods, TAST [24], FAU [69], and CODA (ours). As the batch size varies, CODA consistently delivers superior performance compared TAST and FAU, revealing the advantages of the proposed online adaptation strategy.
Analysis on unknown-awan training objective. We empirically compare dif- ferent unknown-aware training objec- tives discussed in Section 2 and Sec-
Table 4: Analysis on unknown-aware training objective.
Method
PACS
Office-Home Office-31
Digits
Eq. Eq. 2
36.1
34.3
40.7
18.0
tion 3.1, i.e., (q.(1)-(4) and their com- binations. For a fair comparison, we do not involve any TTA strategies including our target disambiguation. The results reported in Table 4. We can observe that our full source compaction is clearly better than its variants, revealing the superiority of our optimization procedures on both real known-class samples and virtual unknown-class samples.
41.1
58.9
63.0
39.3
Eq.
46.8
62.4
72.6
38.6
Eq. 4
43.2
60.3
70.7
37.4
Eq.
3 + Eq. 4
51.2
64.8
75.3
39.0
Analysis on unknown classes. In Figure (c)-(d), we study the impact of varying the number of known classes on three methods, ERM [60], One Ring-S [77]. and CODA (ours). Note that the total number of classes (i.e. IC, J C. remains unchanged. Even when the number of known classes is small, CODA still exhibits superior performance. This advantage remains consistent the number of known classes changes. Consequently, CODA is capable of handling extreme scenarios.
Feature visualization. We use t-SNE [59] to visualize the features of four models Office-31 dataset, i.e., ERM, One Ring-S, Source Compaction, and CODA (full). The results are depicted in Figure 7, where various colors, excluding gray, signify different known classes, and gray points represent unknown classes. It is noteworthy that the embedding features learned by two baselines (ERM and One Ring-S) S) fail to present a clear separation, resulting in ambiguous boundaries among

---

Heggom
SAN
7607
Batch SUM
5222
Koown desses
(a)
(b)
(c)
(d)
Figure 6: (a)-(b) The influence of test batch size. (c)-(d) The influence of varying the number of known classes. Figures (a) and (c) are plotted based on the Office- dataset, while figures (b)
and (d) derived from the PACS dataset.
(a) ERM
(b) One Ring-S [77]
(c) Source Compaction
(d) CODA (full)
Figure 7: t-SNE visualization of the learned features the Office-31 dataset.
different classes particularly between known and unknown ones. Instead, CODA can learn the intrinsic
structure ("manifold") of the target samples, providing more discriminable clustering patterns.
Related Work
Domain Generalization (DG), DG is concerned with the training of a model using data from either multiple single source domain, with the goal of achieving good generalization previously unseen target domains. Mainstream approaches usually involve invariant learning and robust learning with elaborate training objectives. Based the focus of this paper,
classify existing methods into three categories and provide descriptions as follows. (1) Closed-set DG. Existing methods can be roughly divided into four categories: feature matching baasdd based [32, 31, 40, 91, 83), decomposition-
based [46, 45, 13, 39, 54, 36, 78], augmentation-based [65, 85, 86, 74, 41, 88, 8J, and meta-learning- based approaches [29, 33, 30, 37, 7]. (2) Open-set DG. is worth noting that very few works have DG in open-set scenarios. A handful of recent studies [53, 90, 77, 9] started consider the existence of both known and unknown classes in novel DG settings, such as Open-Set DG (OSDG) [90, 77]. For example, Yang et al. [77] hold the view that any category other than the ground-truth can be considered as part of the unknown classes. Chen et al. [9] first activate the unknown's 's logit via an unknown-awand aware training loss and then introduce a test-time adjustment strategy to refine the model prediction. Zhu et al. [90] rely on heuristic thresholding mechanisms to distinguish known- and unknown-che
delved into the exploration
samples. (3) DG by test-time adaptation. According to
Dubey et al. [14], model trained solely on source data will inevitably experience an "adaptivity gap" when it is directly employed on target domains, emphasizing the necessity of on-target adaptation. Grounded on this insight, several recent works [79, 73, 72, 11] resort TTA for mitigating the adaptivity gap, such as adaptive risk minimization [79], energy-based based sample adaptation [72], and improved self-supervised lcarning tasks [11].
Test- Time Adaptation (TTA). TTA [34] is an emerging paradigm that has demonstrated immense potential in adapting pre-trained models unlabeled data during the testing phase. A plethora of approaches [35, 58, 67, 23, 43, 10, 80, 69] have been developed to improve the predictions of source- trained models on target domain with online training/adapttation strategies. TTT [58] introduces self-supervised learning tasks (e.g. rotation classification) to both source and target domains. Tent [67] leverages the batch statistics of the target domain and optimizes the channel-wist affine parameters by entropy minimization. T3A [23] proposes use class prototypes for adjusting predictions and introduces support set to memorize representative and reliable samples. TAST [24] improves T3A by proposing nearest neighbor information induced self-training framework.

---

Detection (OD), OD [75], which seeks to identify novel instances that the model has not encountered during training, close to OTDG setting. Prevailing OD methods center on creating OOD scoring functions, for example, confidence- baseed techniques [4, 20, 21], distance- based scores [27, 51, 57], and energy-hased scores [38, 55J. Although promising. the OD approach limited to binary classification problems and lacks the capability to effectively explore domain shift and open class challenges the test data.
There are also other topics related to open-world machine learning [44] that bear certain relevance to our work, including open-set recognition [50, 62], novel class discovery [15, 61], zero-shot learning [47, 70J, and class-incremest learning [6, 82], to name few. Compared to previous methods, our work addresses two types open-world situations (i.e., domain shift and open class), supporting generalization capabilities consistently throughout the training and inference phases.
6 Conclusion
In this paper, solve the problem of OTDG where both domain shift and open classes may concurrently arise on the unseen test data. We introduce a two-stage framework (CODA) for efficiently learning what don't 't know in the wild. At the training stage, we compact the embedding of source known classes and thus reserve space for target unknown classes. the testing phase, we introduce training objective mitigate the optimality gap between domains while avoiding catastrophic forgetting. Empirically, CODA achieves superior performance on standard DG benchmarks.
Acknowledgement
This work was partially supported by Hong Kong Research Grants Council under Collaborative Research Fund (Project No. HKU C7004-22G).
References
[1] Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In ICML, pages 145-155, 2020.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint / 190700293, 2019.
[3] Shai Ben-David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In International Conference on Artificial Intelligence and Statistics, pages 129- 136, 2010.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR, pages 1563-1572, 2016.
Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. ECCV, 2020.
Francisco M Castro, Manuel J Marin-Jiménezz Nicolás Guil, Cordelia Schmid, and Karteck Alahari. End-to-end incremental learning. ECCV, 2018.
[7] Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaoging Liu, and Yizhou Yu. Compound domain general- ization via meta knowlekn encoding. In CVPR, pages 7119-7129, 2022.
[8] Chaoqi Chen, Luyao Tang. Feng Liu, Gangming Zhao, Yue Huang, and Yizhou Yu. Mix and reason: Reasoning over semantic topology with data mixing for domain generalization. NewrIPS, 35:33302-3355 33315, 2022.
[9] Chaoqi Chen, Luyao Tang, Leitian Tao, Hong- Yu Zhou, Yue Huang, Xiaoguang Han, and Yizhou Yu. Activate and reject: Towards safe domain generalization under category shift. In ICCV, pages 11552-11563, 2023.
[10] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, pages 295-305 305, 2022.
[11] Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In CVPR, 2023.
[12] Noam Chomsky. Aspects of the Theory of Syntax, volume 11. MIT press, 2014.
10

---

[13] Rune Christiansen. Niklas Pfister. Martin Emil Jakobsen, Nicola Gnecco, and Jonas Peters. A causal framework for distribution generalization. IEEE TPAMI, 2021.
[14] Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for real-world domain generalization. CVPR, 2021.
[15] Enrico Fini, Enver Sangineto. Stéphane Lathuiliere, Zhun Zhong, Moin Nabi, and Elisa Ricci, unified objective for novel class discovery. ICCV, 2021.
[16] Bo Fu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In ECCV, pages 567-583. Springer, 2020.
[17] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, pages 1180-1189, 2015.
[18] Boqing Gong, Yuan Shi. Fei Sha, and Kristen Grauman. Geodesic flow kemell for unsupervised domain adaptation. In CVPR, pages 2066-2073. IEEE, 2012.
[19] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778,77 2016.
[20] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
[21] Rui Huang and Yixuan Li. Towards scaling out-of-distribution detection for large semantic space. In CVPR, 2021.
[22] Jonathan J. Hull. A database for handwritten text recognition research. TPAMI, 16(5):50--554 1994.
[23] Yusuke Twasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. NeurlPS, 34:2427-2440, 2021.
[24] Minguk Jang and You-Young Chung. Test-time adaptation via self-training with nearest neighbor informa- tion. ICLR, 2023.
[25] Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, (44):441-551, 1989.
[26] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
[27] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NewrIPS, pages 7167-7177, 2018.
[28] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. ICCV, pages 5542-5550, 2017."
[29] Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI, 2018.
[30] Da Li, Jianshu Zhang, Yongxin Yang. Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, pages 1446-1455, 2019.
[31] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, pages 5400-5409. 2018
[32] Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, pages 624-639, 2018
[33] Yiying Li. Yongxin Yang. Wei Zhou, and Timothy Hospedales. Feature-critic networks for heterogeneous domain generalization. In ICML, pages 3915-3924, 2019.
[34] Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303 15361, 2023.
[35] Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, pages
PMLR, 2020.

---

[36] Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang. Tao Li, Tao Qin, Wei Chen, and Tie-Yan Liu. Learning causal semantic representation for out-of-distribttion prediction. NewrIPS, 34, 2021.
[37] Quande Liu, Qi Dou, and Pheng-Ann Heng. Shape- meta-learning for generalizing prostate mri segmentation unseen domains. In MICCAI, pages 475-485. Springer, 2020.
[38] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based -uttrf-distribution detection. In NeurIPS, 2020.
[39] Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In ICML, pages 7313-7324, 2021.
[40] Toshihiko Matsuura and Tatsuya Harada. Domain generalization using mixture of multiple latent domains. In AAAI, 2020.
[41] Hyconscob Nam, HyunJac Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, pages 8690-8699, 2021.
[42] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[43] Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh Ap. Generalization on unseen domains via inference-time label-preserving target projections. In CVPR, pages 12924-12933, 2021.
[44] Jitendra Parmar, Satyendra Chouhan, Vaskar Raychoudhury, and Santosh Rathore. Open- world machine learning: applications, challenges, and opportunities. ACM Computing Surveys, 55(10):1-37,
2023. 2023.
[45] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. In ICML, pages 7738,, 2020.
[46] Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR, 19(1:1309-134) 2018.
[47] Bernardino Romera- Paredes and Philip Torr. An embarrassingly simple approach to zero-shot lcarning. In ICML, 2015.
[48] Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models new domains. In ECCV, pages 213-226, Springer, 2010.
[49] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In ECCV, pages 153 168, 2018.
[50] Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE TPAMI, 35, 2012.
[51] Vikash Sehwag. Mung Chiang. and Prateek Mittal. Ssd: A unified framework for self supervude outlier detection. ICLR, 2021.
[52] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-baser localization. ICCV, pages 618-626, 2017.
[53] Yang Shu, Zhangjic Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain-augmenter meta-learning. In CVPR, pages 9624 9633, 2021.
[54] Xinwei Sun, Botong Wu, Xiangyu Zheng, Chang Liu, Wei Chen, Tao Qin, and Tic- Yan Liu. Recovering latent causal factor for generalization to distributional shifts. NewrlPS, 34, 2021.
[55] Yiyou Sun, Chuan Guo, and Yixuan Li. React: distt--diittitution detection with rectified activations. In NewrlPS, 2021.
[56] Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribbttio detection. In ECCV, 2022.
[57] Yiyou Sun, Yifci Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In /CML, 2022.
[58] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML, pages 9229-92229 PMLR, 2020.
12

---

[59] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using I-sne. JMLR, 9(11), 2008.
[60] Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions neural networks, 10(5):988 999, 1999.
[61] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In CVPR, 2022.
[62] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: good closed-set classifier is all you need. In ICLR, 2022.
[63] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR, pages 5018-5027, 2017.
[64] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML, 2019.
[65] Riccardo Volpi and Vittorio Murino. Addressing model vulnerability to distributional shifts over image transformation sets. In ICCV, pages 7980-7989, 2019.
[66] Riceardo Volpi, Hongscok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NewrIPS, pages 5334-5344, 2018.
[67] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell, Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
[68] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. TKDE, 2022.
[69] Shuai Wang, Danan Zhang. Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In CVPR, 2023.
[70] Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10())1137, 2019.
[71] Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng. Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICMI., 2022.
[72] Zehao Xiao, Xiantong Zhen, Shengcai Liao, and Cees GM Snock. Energy- based test sample adaptation for domain generalization. In ICLR, 2023.
[73] Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees GM Snoek. Learning to generalize across domains on single samples. In ICLR, 2022.
[74] Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, and Mare Nicthammer. Robust and generalizable visual representation learning via random convolutions. In ICLR, 2021.
[75] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized outto--diitribuution detection: survey. arXiv preprint arXiv:
2/ 10. 1134, 2021.
[76] Shiqi Yang, Yaxing Wang, Kai Wang, Shangling Jui, et al. Attracting and dispersing: A simple approach for source-free domain adaptation. In NeurlPS, 2022.
[77] Shiqi Yang, Yaxing Wang, Kai Wang, Shangling Jui, and Joost van de Weijer. One ring to bring them all: Towards open-set recognition under domain shift. arXiv preprint vrXiv:2206.036000 2022.
[78] Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf. and Eric P Xing. Towards principled disentanglement for domain generalization. In CVPR, pages 8024-8034, 2022.
[79] Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. NeurIPS, 34:23664-23677 2021.
[80] Marvin Mengxin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In NewrIPS, 2022.
[81] Long Zhao, Ting Liu. Xi Peng, and Dimitris Metaxas. Maximum-entron entropp adversarial data augmentation for improved generalization and robustness. NewrIPS, 33:14435 111144435-1447,,, 2020.
13

---

[82] Da- Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shillang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In CVPR. 2022.
[83] Hong- Yu Zhou, Yizhou Yu, Chengdi Wang. Shu Zhang, Yuanxu Gao, Jia Pan, Jun Shao, Guangming Lu, Kang Zhang, and Weimin Li. A transformer-bseed pepnesentation-earniin model with unified processing of multimodal input clinical diagnostics. Nature Biomedical Engineering, 7:743-755, 2023.
[84] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. TPAMI, 2022.
[85] Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation domain generalisation. AAAI, pages 13025-13032, 2020.
[86] Kaiyang Zhou, Yongxin Yang. Timothy Hospedales, and Tao Xiang. Learning generate novel domains domain generalization. In ECCV. pages 561-578, 2020.
[87] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. TIP. 30:8008-8018, 2021.
[88] Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR, 2021.
[89] Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In ICML, pages 27222-27244, 2022.
[90] Ronghang Zhu and Sheng Li. Crossmatch: Cross-classifier consistency regularization for open-set single domain generalization. ICLR, 2022.
[91] Wei Zhu, Le Lu, Jing Xiao, Mei Han, Jiebo Luo, and Adam Harrison. Localized adversarial domain generalization. CVPR, pages 7108-7118 18, 2022.
14


Reviews:
[{"review_id": "BXHJFun1yL", "reviewer": "Reviewer_bs2E", "summary": "This paper introduces a novel problem formulation termed Open Test-Time Domain Generalisation, which accounts for the presence of both domain shift and unseen classes during the testing phase. To tackle this problem, a two-stage methodology is proposed, comprising Compaction and Disambiguation. The primary objective of the Compaction stage is to facilitate the creation of a region in the embedding space that can accommodate the incoming unknown classes at test-time. This is accomplished by introducing synthetic unknown classes into the latent space by compacting the embedding space associated with the known classes from the source domain.  At training time, in lieu of unknown class data, manifold mixup is used to synthesise the unknown class training data and a classifier is trained that has additional decision boundaries corresponding to the unknown classes.\n\nWhile the compaction stage is proposed during training to accommodate new classes, the disambiguation phase is proposed to handle the potential domain shift in the target domain. For the disambiguation phase, authors adopt existing TAST [ref 20 from paper] methodology and apply it to a setting with unknown classes.\n\nExperimental results show that  the current method has been exhaustively tested against baselines. ", "questions": "* While the statement “...we activate the response of the real known class towards unknowns (Eq. 3)” from line 124 is adequately explained, we believe that further explanation of line 125, “... and the response of the virtual unknown class towards the known ones” is required.\n* In line 111, y_hat shouldn’t be the index but rather a one-hot vector with non-zero entry at the index that y_hat currently holds. Similarly with line 121, “ y’ ” is incorrectly defined.\n* In Eq.3 and Eq.4, a new undefined term ‘C’ is introduced.\n* We request the authors to use notation consistent with TAST [ref 20 from paper] when building upon their contributions, for increased accessibility.\n* Please justify the choice of scoring function in line 165.\n* In line 185, the unknown class set cannot include the source domain class set.\n* Line 205 requires clarification.\n* While Eq.5 shows that the prototype-based classifier assigns 'z' to one of |C_s|+N classes, line 60 of the supplementary file contradicts that by assuming all N classes to be one class. Please clarify.\n", "limitations": "None in particular", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "+ Novel Problem Formulation: The paper introduces a novel problem formulation called Open Test-Time Domain Generalization, which addresses the challenges of domain shift and unseen classes during the testing phase. This formulation extends the scope of traditional domain generalization methods and accounts for realistic scenarios where new classes can emerge at test time.\n+ Integration of Manifold Mixup: To address the lack of unknown class data during training, the paper leverages manifold mixup to synthesize the unknown class training data. This integration allows the model to learn decision boundaries corresponding to the unknown classes. By utilizing this data augmentation technique, the proposed method enhances the model's ability to generalize to unknown classes and improve overall performance.\n+ Extensive Experimental Evaluation: The paper demonstrates the thoroughness of the research by subjecting the proposed method to exhaustive testing against relevant baselines.\n", "weaknesses": "The work brings together ideas from different existing methods and provides a novel way to tackle the problem setting of Open Test-Time Domain Generalisation. While there are a certain amount of novel claims made in the paper, they are often unsubstantiated. We have provided some pointers to where further explanations are required. Additionally, we recommend that paper be thoroughly proofread.\n\n* The paper lacked a sufficient level of explicitness in conveying the necessary information. Important details or background knowledge seemed to be assumed or left to inference, which makes it challenging for readers to follow the logical flow of the study. It would greatly benefit the clarity of this work if additional context, explanations, and descriptions could be provided to ensure a more straightforward understanding for readers who may not possess the same background knowledge or expertise as the authors. For instance, (a)\nWhile Eq.5 shows that the prototype-based classifier assigns 'z' to one of |C_s|+N classes, line 60 of the supplementary file contradicts that by assuming all N classes to be one class. Please clarify. (b) The mechanics of support set, its constitution and updation are ill-defined.\n* While the paper asserts the mitigation of catastrophic forgetting in line 16, it lacks empirical evidence to validate this claim. Therefore, it is recommended to include relevant results in order to substantiate the effectiveness of the proposed approach in addressing catastrophic forgetting. Reporting ablation study of semantic consistency loss on source domain known classes is advised.\n* In line 133, the inequality seems to be incorrect and the assertion in lines 134-135 does not trivially follow from the inequality. \n* Lines 132 through 136 introduce the notion of optimality gap which we believe is a misnomer as this is defined as “adaptivity gap” [ref 11 from paper]. However, regardless of the appropriateness of the term optimality gap, we believe that the problem that TTA resolves is domain shift, rendering the previous lines irrelevant.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer for the positive feedback and insightful comments! We extend our apologies for any inconvenience or misunderstanding that may have occurred when reading our manuscript, and we assure you that we will incorporate all of your valuable suggestions in our final version. Our responses to your specific questions are listed below.\n\n> **Q1. While Eq.5 shows that the prototype-based classifier assigns 'z' to one of |C_s|+N classes, line 60 of the supplementary file contradicts that by assuming all N classes to be one class.**\n\nWe apologize for the confusion. The statement in line 60 of the supplementary file is indeed imprecise. What we intended to convey is that our method aims to classify unknown classes into various virtual unknown classes. These can essentially be regarded as a whole, as we did not pursue the precise clustering of the unknown classes themselves. We will deliver this point more precisely in the revised manuscript.\n\n> **Q2. The mechanics of support set, its constitution and updation are ill-defined.**\n\nThe constitution and updation of the support set (aka. memory bank) were defined in lines 141-155 (main text). To facilitate a better understanding, we highlight several key aspects: (1) the support set consists of two types of sources: penultimate features and logits; (2) for each test sample, we compute its nearest neighbor from the support set; (3) the local class conditionals are estimated from the nearest neighbor set (Eq. 6); (4) the estimated local class conditionals are used to update the global class prototype of the support set (Eq. 7) in a moving-average style.  \n\n> **Q3. The effectiveness of the proposed approach in addressing catastrophic forgetting. Reporting ablation study of semantic consistency loss on source domain known classes is advised.**\n\nNice question! As suggested, we report the ablation of semantic consistency loss on source domain known classes in the newly attached PDF document. As depicted in Fig. 1 of the document, we can observe a noticeable decline in source domain accuracy as testing proceeds when the semantic consistency constraint is removed. In contrast, by ensuring semantic consistency between the target and original source models, our approach ensures robustness throughout the test-time adaptation process.\n\n> **Q4. In line 133, the inequality seems to be incorrect and the assertion in lines 134-135 does not trivially follow from the inequality.**\n\nCorrection: $\\varepsilon_S(h^t)<\\varepsilon_T(h^*)$ --> $\\varepsilon_T(h^t)<\\varepsilon_T(h^*)$. We are sorry for the lack of carefulness. This inequality represents that the target optimal classifier $h^t$ will perform better than the universal optimal classifier $h^*$ on the target domain $T$. Consequently, in lines 134-135, we conclude that ''it is not feasible to find a universal optimal classifier that applies to both source and target domains'', highlighting the importance of our test-time adaptation module.\n\n> **Q5. Lines 132 through 136 introduce the notion of optimality gap which we believe is a misnomer as this is defined as “adaptivity gap” [ref 11 from paper].**\n\nThe authors of [11] conceptualized ''adaptivity gap'' in the context of DG but did not provide a detailed mathematical definition. We formalize this issue and rename it as ''optimality gap'' to reflect its inner characteristics. To keep the consistency, we will adhere to the naming used in [11] (i.e. adaptivity gap) in the final version.\n\n> **Q6. Further explanation of line 125, “... and the response of the virtual unknown class towards the known ones” is required.**\n\nAs shown in Figure 1, Eq. (3) and Eq. (4) are symmetric. In Eq. (3), real known-class samples will give the largest response to its ground-truth (known-class) dimension, and the second-largest response to a certain unknown-class dimension. Similarly, In Eq. (4), virtual unknown-class samples will give the largest response to its ground-truth (unknown-class) dimension (first term), and the second-largest response to a certain known-class dimension (second term). \n\n> **Q7. In line 111, y_hat shouldn’t be the index but rather a one-hot vector with non-zero entry at the index that y_hat currently holds. Similarly with line 121, “ y’ ” is incorrectly defined.**\n\nWe apologize for this oversight. You are absolutely correct. To align with Eq. (1) and Eq. (2), we revise the second term of Eq. (3) and Eq. (4) as:\n\n* $-\\log \\frac{ \\exp (f_{\\hat{y}}(\\mathbf{x}))}{\\sum_{c\\in |\\mathcal{C}_s|+N, c \\neq y} \\exp (f_c(\\mathbf{x}))}$\n* $-\\log \\frac{ \\exp (h_{y'}(\\hat{\\mathbf{z}}))}{\\sum_{c\\in |\\mathcal{C}_s|+N, c \\neq \\hat{y}} \\exp (h_c(\\hat{\\mathbf{z}}))}$\n\n> **Q8. In Eq.3 and Eq.4, a new undefined term ‘C’ is introduced.**\n\nSorry for the typo. The 'C' here corresponds to 'N', i.e.,  the number of virtual class prototypes defined in line 108.\n\n> **Q9. We request the authors to use notation consistent with TAST [ref 20 from paper] when building upon their contributions, for increased accessibility.**\n\nThank you for your suggestion. We will revise our manuscript to align notations with [20].\n\n> **Q10. Please justify the choice of scoring function in line 165.**\n\nEq. (10) is designed to exclude unreliable samples as determined by their entropy during test-time adaptation, and it gives more weight to reliable samples that have lower prediction uncertainties. By doing so, we ensure that those reliable samples exert a greater influence during the test-time model updating.\n\n> **Q11. In line 185, the unknown class set cannot include the source domain class set.**\n\nSorry for the typo. 'combined with $\\mathcal{C}_s$' should be removed.\n\n> **Q12. Line 205 requires clarification.**\n\nFollowing [82], a target sample is marked as an unknown class if its entropy is larger than a threshold, which is empirically set to $\\log(|\\mathcal{C}_s|)/2$ where $|\\mathcal{C}_s|$ represents source label space and $\\log(|\\mathcal{C}_s|)$ is the maximum value of entropy. "}, {"review_id": "Db9F4BiFrG", "reviewer": "Reviewer_BmaN", "summary": "This paper focuses on the problem of domain adaptation. In particular, in addition to the already-studied setting of domain shift at test time, this paper considers the practical scenario of unknown classes being presented at test time. Often current models will incorrectly predict a known class label for examples from unknown classes. This paper addresses this by: Compaction and Disambiguation. Compaction forces the known classes embeddings to be more compacted, with regions in between classes being reserved for unknown classes through the introduction of virtual unknown classes. Disambiguation, done at test time, is for test time adaptation. The objective in this phase optimizes 1) consistency between model output predictions using a nearest neighbor memory bank of class prototypes 2) consistency between source and target model to avoid catastrophic forgetting. ", "questions": "Can the authors explain in more detail why the known-class examples must activate unknown-class logits?", "limitations": "The authors could include a study on how the test time adaptation affects source domain accuracy i.e. what is it before and after the disambiguation protocol? I'm not sure if these results have been included or discussed thoroughly. Since the TTA loss has a term to encourage consistency with source model, how good the model is at this objective should be evaluated. ", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) Introduction of a novel problem: unknown classes at test time in addition to domain shift \n\n2) Thorough ablation on various components of methods to illustrate why every choice is necessary\n\n3) Strong empirical results (SOTA across several datasets - I am not too familiar with what the standard datasets for domain generalization are though)\n", "weaknesses": "1) More background on test-time adaptation can be given before introducing the \"disambiguation\" protocol since many readers (like myself) may not be very familiar with this\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks very much for your insightful comments and suggestions! We have properly revised our paper based on your reviews and below are our detailed responses.\n\n> **Q1. More background on test-time adaptation can be given before introducing the \"disambiguation\" protocol.**\n\nWe appreciate the reviewer's suggestion. Test-Time Adaptation (TTA) is an emerging paradigm that aims to adapt a source-trained model to novel target environments by progressively tuning the model weights using unlabeled test data during inference time. Compared to the traditional domain generalization pipeline, TTA mitigates the potential optimality gap (see Definition 1) and thus makes the whole process more practical for real-world applications. For example, suppose that we wished to deploy modern vision systems in an autonomous vehicle or a robot to recognize objects in the wild. Given that test environments can unpredictably encounter new objects at different stages, it is inefficient and even impossible to collect a set of labeled data to retrain the model. TTA exactly addresses this challenge, presenting a promising solution. Despite the promise, state-of-the-art TTA methods struggle to handle the open class as they typically seek the complete semantic alignment (i.e. cross-domain one-vs-one alignment) between source and target classes, resulting in the ''ambiguation'' between known and unknown classes (i.e. the source and target label spaces are asymmetric). We have integrated the aforementioned discussion into the revised manuscript for greater clarity. \n\n> **Q2. Why the known-class examples must activate unknown-class logits?**\n\nIn our setting, we have no access to real unknown-class examples during the training phase. Consequently, we propose to use real known-class examples (second term in Eq. 3) and virtual unknown-class examples (first term in Eq. 4) to activate unknown-class logits. Empirically, as shown in Table 4, the prediction performance will significantly degenerate when only using Eq. 4. Here, we provide two perspectives to interpret why the known-class examples must activate unknown-class logits (second term in Eq. 3). **(1)** The response region of unknown classes (i.e., the gray regions in Fig. 2) is positively correlated to the size of unknown-class logits. Therefore, activating the unknown-class logits of known-class examples helps to enlarge the response region of unknown classes, thereby enhancing the unknown-aware ability of the source-trained model. **(2)** From the other perspective, since the virtual unknown classes are synthesized from known-class examples, activating the unknown-class logits of known-class examples will make the data manifold more smooth. As shown in Fig. 2, the virtual unknown-class samples (i.e., dashed points) compact the space of known-class samples (Fig. 2(b) vs. Fig. 2(c)), making the high-density regions of virtual unknown classes and known classes much closer and evenly distributed. This, in turn, smoothens the boundary delineations between known and unknown classes.\n\n> **Q3. The authors could include a study on how the test time adaptation affects source domain accuracy i.e. what is it before and after the disambiguation protocol?**\n\nThank you for pointing out this crucial and often overlooked issue. We have added the results regarding how the TTA affects source domain accuracy in the newly attached PDF document. As depicted in Fig. 1 of the document, we can observe a noticeable decline in source domain accuracy as testing proceeds when the semantic consistency constraint is removed. In contrast, by ensuring semantic consistency between the target and original source models, our approach ensures robustness throughout the test-time adaptation process."}, {"review_id": "Fl3JQ1FyBp", "reviewer": "Reviewer_nkce", "summary": "This research paper introduces a novel experimental configuration referred to as Open Test-time Domain Generalization (OTDG), which can be viewed as an amalgamation of domain generalization, test-time adaptation, and open-set recognition. In order to address this particular experimental configuration, the authors put forward a fresh conceptual framework termed Compaction and Disambiguation (CODA). The fundamental concept behind this framework entails the imposition of restrictions on decision boundaries during the training phase, leveraging labeled source data, with the aim of ensuring an ample latent space to accommodate future classes. Subsequently, these decision boundaries are further refined during the test phase through the utilization of unlabeled target data.", "questions": "As I said above, can the authors list some practical scenario of the OTDG setup and convince me this is not a hand made setup just for research.", "limitations": "One of the main limitations of this study revolves around the practicality of the proposed setup. It is natural to question whether the setup accurately reflects real-world scenarios and if it can be applied in practical contexts. To gain more insight into the effectiveness and applicability of the Open Test-time Domain Generalization (OTDG) problem, it would be valuable to explore realistic scenarios that closely resemble the challenges faced in practical settings. This would help establish a stronger connection between the proposed approach and its potential real-world applications.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, The experimental setting introduced in this study represents a novel paradigm. Although its practical viability is yet to be established, it presents a unique problem that has not been previously encountered.\n\n2, The paper effectively presents a well-structured argument, providing a reasonable proposition of loss functions aimed at addressing the OTDG problem.\n\n3, The experiments conducted in this study are meticulously executed and accompanied by illustrative figures that effectively convey the findings. The evaluation involves a comprehensive comparison with recent state-of-the-art works that are closely related to the subject matter, ensuring a robust analysis of the proposed approach.", "weaknesses": "The motivation behind the concept of Open Test-time Domain Generalization (OTDG) in this study may benefit from further clarification. It would be valuable to present realistic scenarios that exemplify the challenges addressed by this setup. By providing concrete examples or case studies that demonstrate the practical relevance of OTDG, the study can establish a stronger connection between the proposed approach and its potential real-world applications. This would enhance the understanding of how OTDG can be applied in practical scenarios and highlight its significance in addressing specific challenges faced in domains that require generalization, test-time adaptation, and open-set recognition. The proposed solution in this study demonstrates a reasonable level of novelty in addressing the challenges posed by the specific OTDG problem. While it is acknowledged that there may be similarities with previous related works, this is considered a minor aspect. The study's main focus lies in presenting a solution that effectively tackles the challenges unique to the OTDG problem, highlighting the authors' contribution in the field. By emphasizing the novel aspects of the proposed solution and its potential impact, the study provides valuable insights and advancements within the context of the OTDG problem.\n\nMinor weakness: no codes in the supplementary material.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive feedback and for acknowledging the innovative aspects of our study. We address the concerns as follows and will improve the manuscript accordingly.\n\n> **Q1. List some practical scenarios of the OTDG setup.**\n\nGreat question! We would like to discuss three practical scenarios of OTDG: \n\n**(1)** Suppose we aim to integrate modern vision systems into autonomous vehicles. When shifts are limited to the environment (such as weather and illumination) or the appearance (such as size and viewpoint) of previously observed objects, current state-of-the-art techniques can correct for the potential shifts on the fly, such as Domain Adaptation (DA) and Domain Generalization (DG). But what if the sudden arrival of *new* objects in an *ever-changing* world? Most existing approaches will break and are likely to result in catastrophe, raising strong concerns about model safety and reliability. This brings two new challenges: 1) open and ever-changing environments hinder the usage of traditional DA (need access to target data for joint training with source domain) and DG (face significant performance degradation due to the open class and adaptivity gap) methods, and 2) test-time adaptation offers a promising direction but lacks principled solutions for addressing the open class in the context of DG. To this end, the proposed OTDG setup and CODA framework align precisely with this direction and provide compelling empirical results.\n\n **(2)** When deploying a machine learning system tailored to diagnose diseases based on clinical symptoms, the underlying pre-assumption is that the data distribution remains stable over time. However, diseases don't always follow the IID and closed-set assumptions, given their ever-evolving nature. For example, the situation becomes dicey when we are blindsided by novel diseases, such as COVID-19 and its successive variants.\n\n **(3)**  Let's further consider the application of machine learning systems in industrial anomaly detection. Typically, these systems are trained on historical data, assuming that machine operations and associated data patterns remain consistent. Using this data, techniques like Anomaly Detection (AD) are employed to identify and rectify abnormal behaviors. However, in a rapidly modernizing industrial landscape, novel machinery and processes are introduced frequently. What happens when an unprecedented anomaly, resulting from these new introductions, manifests? Traditional models might fail to detect or even misclassify such anomalies, potentially causing equipment damage or production halts. \n\n> **Q2. Minor weakness: no codes in the supplementary material.**\n\nThank you for raising this point. We intend to release the code upon the paper's acceptance."}, {"review_id": "DyMZ4VerfI", "reviewer": "Reviewer_shJ8", "summary": "This paper proposes a two-stage framework for the task of open-class domain generalization. During training, it introduces virtual unknown classes for the learning objective to insert unknowns into the latent space by compacting the embedding space of known classes. During the test phase, it proposes a prototype-based test-time adaptation pipeline, which involves (1) consistency between the predicted class distributions and the estimated class conditionals; (2) class-wise sample reliability for ensuring the quality of target pseudo labels; (3) semantic consistency between the source-trained model and target model, to disambiguate the decision boundaries between known and unknown classes. Experimental results on different DG benchmarks demonstrate its effectiveness.", "questions": "See weakness.", "limitations": "Limitations are discussed in the supplementary material.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper studies a practical problem that has been rarely explored, open test-time domain generalization, which considers the co-occurrence of domain shift and open class. The comparison with other related settings is clearly stated and provides good intuitions.\n\n2. The proposed framework, CODA, is novel and technically sound. The introduced compaction and disambiguation process is easy to implement, and the computational overhead is relatively small. Overall, the paper is well organized, and the advantage of CODA is clearly demonstrated.\n\n3. The experiments confirm the effectiveness of the method. In particular, different types of baselines, such as DG, DA, TTA, and OOD detection, are explicitly compared. Ablations and visualization further justify the contributions of individual components.", "weaknesses": "1. There are some inconsistencies that need to be clarified. Firstly, the authors initially utilize the term 'adaptivity gap', but introduce a new terminology, 'optimality gap', in Definition 1. What are their distinction and correlation? Secondly, the proposed setting called OTDG appears to underscore the process of test-time operation. However, I found that the source training stage offers large improvements according to the reported results. As such, I question whether the use of 'test-time domain generalization' is fitting. Thirdly, in Figure 2, the authors use dashed points to signify virtual unknown samples. However, these samples are produced by manifold mixup at the feature level, which makes them distinct from the original data.\n\n2. Although Figure 1 provides an intuitive comparison among different learning objectives, I struggle to fully comprehend why Equation (3) is superior to Equation (2). The advantages of employing multiple virtual prototypes versus a single virtual prototype warrant further discussion and comparison. \n\n3. Basically, the target disambiguation is a prototype-based classification process. As the target distribution contains some open classes, how the disambiguation is done by only using the source-learned virtual prototypes? Moreover, could the proposed CODA be extended to a more challenging scenario that the open classes are further classified into different clusters?\n\n4. How are the experiments conducted with different types of baselines, including aspects like train-validation split and model selection? Further clarification and details regarding the fairness of comparison should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback and for recognizing our novel contributions. We list our responses to your questions below.\n\n> **Q1. The authors initially utilize the term 'adaptivity gap', but introduce a new terminology, 'optimality gap', in Definition 1. What are their distinction and correlation?**\n\nThe term 'adaptivity gap' was first introduced in [11]. In this work, we define a new terminology - optimality gap - to highlight the disparity between the target optimal classifier and the universal classifier. In essence, these two terminologies can be used interchangeably. For the sake of consistency, we will adopt the terminology from [11] in the final version.\n\n> **Q2. The proposed setting called OTDG appears to underscore the process of test-time operation. However, I found that the source training stage offers large improvements according to the reported results. As such, I question whether the use of 'test-time domain generalization' is fitting.**\n\nThank you for pointing out this interesting problem. In this work, we focus on DG safety by solving two open-world issues in a unified framework: open class and adaptivity gap. The former is solved by unknown-aware training objectives, while the latter is addressed by test-time adaptation. These two components are complementary to each other and should not be regarded separately. We agree that the source training stage offers large improvements according to the reported results, but these empirical results do not affect the ultimate goal of OTDG.\n\n> **Q3. In Figure 2, the authors use dashed points to signify virtual unknown samples. However, these samples are produced by manifold mixup at the feature level, which makes them distinct from the original data.**\n\nWe apologize for the confusion. You're correct; the dashed points represent virtual unknown samples created through manifold mixup. We will clarify this in the figure's caption and provide a more detailed description in the main text to avoid future misunderstandings.\n\n> **Q4. I struggle to fully comprehend why Equation (3) is superior to Equation (2). The advantages of employing multiple virtual prototypes versus a single virtual prototype warrant further discussion and comparison.**\n\nThank you for raising this concern. Equation (3) leverages multiple virtual prototypes, which aim to provide a more comprehensive representation of the unknown-class manifold, capturing a wider range of potential variations. On the other hand, Equation (2) with a single virtual prototype may not sufficiently represent the diversity of unknown classes. In our experiments (see Table 4), we found that using multiple virtual prototypes often leads to more robust performance, especially in scenarios with a high diversity of unknown samples, such as PACS and Office-31. We will expand upon this reasoning and include additional comparisons in the revised manuscript to further elucidate the benefits of employing multiple virtual prototypes.\n\n> **Q5. How the disambiguation is done by only using the source-learned virtual prototypes?**\n\nThe source-learned virtual prototypes serve as representative points in the feature space for unknown classes. In the case of OTDG, these prototypes help in identifying and differentiating between the known and unknown samples. Technically, during testing, the distance (i.e. Eq. 5) between a test sample's feature representation and these virtual prototypes can be computed. After that, we could discriminate the real known and unknown classes, i.e., disambiguation.\n\n> **Q6. Could the proposed CODA be extended to a more challenging scenario that the open classes are further classified into different clusters?**\n\nThe CODA framework can indeed be extended to handle scenarios where open classes are further classified into distinct clusters. Leveraging our plug-and-play 'source compaction' and 'target disambiguation' modules, alongside the use of multiple virtual prototypes, provides a robust foundation. These prototypes can potentially serve as initial cluster centers, allowing for further classification of unknown samples. While our current design doesn't directly address this, the foundational elements for such an extension are firmly in place.\n\n> **Q7. How are the experiments conducted with different types of baselines, including aspects like train-validation split and model selection? Further clarification and details regarding the fairness of comparison should be provided.**\n\nOur experiments, including train-validation splits and model selection, strictly adhere to the protocols established by previous OSDG works [70, 82]. We ensure a fair comparison by consistently applying these standards across different types of baselines. This guarantees that any observed performance differences stem from the methods themselves."}]

Author rebuttal:
We thank all the reviewers for their time, insightful suggestions, and valuable comments. We are glad that **ALL** reviewers appreciate our work and find our method simple and effective, highlighting that a novel problem (bs2E, BmaN, nkce, shJ8) - Open Test-Time Domain Generalization (OTDG), a novel and technically sound framework (shJ8) - CODA, extensive experimental evaluation (bs2E, BmaN, nkce, shJ8), strong empirical results (BmaN), and thorough ablation (BmaN, shJ8). We are also encouraged that reviewers find the paper is well organized (shJ8) and presents a well-structured argument (nkce).

We respond to each reviewer's comments in detail below. We have incorporated the reviewers' suggestions into our manuscript, which we believe has significantly strengthened the paper. The main changes we made include:

* We report the source domain accuracy w/ and w/o semantic consistency loss (as Reviewer bs2E and Reviewer BmaN suggested).
* We provide clarifications and corrections to address the reviewer's questions about the main text and have rigorously proofread the entire manuscript (as Reviewer bs2E suggested).
* We provide more background regarding TTA and clarify the motivation for using the known-class examples to activate unknown-class logits (as Reviewer BmaN suggested).
* We list some practical scenarios of the OTDG setup (as Reviewer nkce suggested).
* We offer detailed explanations and clarifications on certain technical aspects (as Reviewer shJ8 suggested).

Meta-review: The paper introduces Open Test-Time Domain Generalization (OTDG), a new concept to handle both domain shifts and unseen classes during testing. The approach has two stages: Compaction, which uses manifold mixup to prepare for unknown classes, and Disambiguation, which deals with domain shifts. Initially, the major concern is that it lacks clarity in some areas and could benefit from further validation and context.

The reviewers are all satisfied with the authors' response and recommend accept. AC reads the paper (roughly), the review, and the discussion, and recommends accept (spotlight).
Decision: Accept (spotlight)