Title: CODA: Generalizing to Open and Unseen Domains with Compaction and Disambiguation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Domain generalization

domain shift

open class

source compaction

target disambiguation
Abstract: The generalization capability of machine learning systems degenerates notably when the test distribution drifts from the training distribution. Recently, Domain Generalization (DG) has been gaining momentum in enabling machine learning models to generalize to unseen domains. However, most DG methods assume that training and test data share an identical label space, ignoring the potential unseen categories in many real-world applications. In this paper, we delve into a more general but difficult problem termed Open Test-Time DG (OTDG), where both domain shift and open class may occur on the unseen test data. We propose Compaction and Disambiguation (CODA), a novel two-stage framework for learning compact representations and adapting to open classes in the wild. To meaningfully regularize the model's decision boundary, CODA introduces virtual unknown classes and optimizes a new training objective to insert unknowns into the latent space by compacting the embedding space of source known classes. To adapt target samples to the source model, we then disambiguate the decision boundaries between known and unknown classes with a test-time training objective, mitigating the adaptivity gap and catastrophic forgetting challenges. Experiments reveal that CODA can significantly outperform the previous best method on standard DG datasets and harmonize the classification accuracy between known and unknown classes.

CODA: Generalizing to Open and Unseen Domains with Compaction and Disambiguation

Abstract

The generalization capability of machine learning systems degenerates notably when the test distribution drifts from the training distribution. Recently, Domain Generalization (DG) has been gaining momentum in enabling machine learning models to generalize to unseen domains. However, most DG methods assume that training and test data share an identical label space, ignoring the potential unseen categories in many real-world applications. In this paper, we delve into a more general but difficult problem termed Open Test-Time DG (OTDG), where both domain shift and open class may occur on the unseen test data. We propose Compaction and Disambiguation (CODA), a novel two-stage framework for learning compact representations and adapting to open classes in the wild. To meaningfully regularize the model's decision boundary, CODA introduces virtual unknown classes and optimizes a new training objective to insert unknowns into the latent space by compacting the embedding space of source known classes. To adapt target samples to the source model, we then disambiguate the decision boundaries between known and unknown classes with a test-time training objective, mitigating the adaptivity gap and catastrophic forgetting challenges. Experiments reveal that CODA can significantly outperform the previous best method on standard DG datasets and harmonize the classification accuracy between known and unknown classes.

1 Introduction

The ability to generalize to unseen environments is considered a key signature of human intelligence [12]. While deep neural networks have achieved great success in many machine learning problems, they are brittle to distribution shifts between training and test domains, which often occur in real-world applications. For example, when deploying object recognition systems in autonomous vehicles, the ever-changing weather conditions ( e.g. fog, rain, and snow) may deteriorate the performance and raise concerns about their reliability over time. This motivates a challenging scenario named Domain Generalization (DG) [68, 84], which extrapolates learning machines to related yet previously unseen test domains by identifying the common factors from available source data.

From the perspective of representation learning, the mainstream paradigm for DG includes invariant risk minimization [2, 1, 89], domain alignment [31], feature disentanglement [45, 36, 78], metalearning [29, 30], and augmentation-based invariant prediction [66, 74, 88]. In spite of the significant progress in DG, the adaptivity gap [14] between source and target domains naturally exists and emerges as an inevitable challenge. Therefore, some prior efforts [23, 24, 72, 11] strive to adapt

the source-trained model through the lens of test-time adaptation [34], which uses unlabeled target samples in an online manner. While these methods alleviate the adaptivity gap to some extent, there is still no guarantee for any specific domain, especially if the domain divergence is large [3]. Moreover, most existing DG methods are designed with the assumption that the label space in the source and target domain are identical, which is too restrictive to be satisfied in practice.

To enable machine learning systems to be resilient to an open world, we aim at a more practical yet under-explored problem named Open Test-time Domain Generalization (OTDG), where both domain shift and open class occur in the unseen target data. The primary challenge of the proposed problem lies in addressing two critical aspects: (1) model generalization with an incomplete training label space, and (2) online model adaptation with asymmetric test label space.

Motivated by this, we propose a novel framework for OTDG, termed Compaction and Disambiguation (CODA). Our key idea is to enforce constraints on the decision boundaries during training using labeled source data and refine them during test time using unlabeled target data. To regularize the model's decision boundary and make the model expandable, CODA introduces a set of virtual unknown classes and optimizes a novel training objective in conjunction with the standard cross-entropy loss. The response to known- and unknown-class logits will be activated for both real and synthesized samples. This process embeds unknowns into the latent space by compacting the embedding space of known source classes, thereby reserving sufficient latent space for target unknown classes. To disambiguate the decision boundaries between known and unknown classes, we propose a novel prototype-based test-time adaptation pipeline. Specifically, the test-time classification should be subjected to three constraints: (1) consistency between the predicted class distributions and the estimated class conditionals, (2) class-wise sample reliability for ensuring the quality of target pseudo labels, and (3) semantic consistency between source-trained and present model predictions.

The main contributions are summarized as follows:

We propose CODA, a simple and effective DG framework to mitigate domain shift and identify open classes in unseen test environments.

We introduce a virtual unknown optimization process to make the model expandable for open classes, and a test-time training objective to match the real test data to corresponding known and unknown class patterns.

We conduct extensive experiments and demonstrate that CODA outperforms previous methods on a series of DG benchmarks.

2 Preliminaries

Problem setup. Let us formally define the OTDG problem. We have access to a source domain D s = { ( x i s , y i s ) } n s i =1 of n s labeled data points and multiple unseen target domains D t = { ( x j t ) } n t j =1 of n t unlabeled data points. Let C s and C t be the source and target class sets, respectively. In OTDG, we have C s ⊂ C t and C u t = C t \ C s is referred to as unknown classes. Assume that X is the input space, Z is the latent space, and Y is the output space. The predictor f = h ◦ g is comprised of a featurizer g : X ↦→ Z that learns to extract embedding features, and a classifier h : Z ↦→ Y that makes predictions based on the extracted features. The goal of OTDG is to find a predictor f : X ↦→ Y that generalizes well to all unseen target domains. Although the labeling function h t is unknown, we assume that we have access to unlabeled instances from D t at test time. In addition, we present the comparison of different problem settings in Table 1.

Unknown-aware training. To solve OTDG, a simple baseline is to train a ( |C s | +1)-way classifier [90, 77, 9], where the additional dimension is introduced to identify the unknown. Formally, we define the standard cross-entropy loss as:

where f ( x ) ∈ R |C s | +1 denotes the network's logit and f k ( x ) is the k -th element of f ( x ) corresponding to the ground-truth label y . As shown in Figure 1 (a), however, such optimization fails to activate the network's response to unknown classes. In this work, we first introduce a simple yet very effective baseline [9] for OTDG. Their key idea is to directly activate the unknown's logit by optimizing its

(a)

(b)

(c)

likelihood, without affecting the ground-truth classification. For a source sample ( x s , y s ) ∈ D s , we minimize the negative log-likelihood w.r.t. unknown logit to increase the unknown probability,

̸

where f u ( x ) is the unknown's logit. Eq. 2 makes the unknown class probability respond to any input sample, irrespective of its class label. Since the learning process is driven by the cross-entropy loss related to the ground-truth category, Eq. 2 does not hurt the performance of known classes.

3 Proposed Method

We propose CODA (Figure 2), a simple two-stage OTDG framework for discovering unknown classes with the help of known ones. The specific implementation of the two stages is described as follows.

3.1 Training-Time Source Compaction

The source compaction stage preserves sufficient space for the upcoming unknown classes without using real target data during training. Since we have no a priori knowledge or assumptions about the characteristics ( e.g., number and attribute) of unknown classes, it is prohibitively difficult to learn meaningful representations beforehand. To begin with, we make a mild assumption that the unknown classes should be far away from all known classes in the embedding space (category shift). In practice, there are two choices for accommodating these unknown classes: (1) low-density region (Figure 1 (b)), and (2) between-class region (Figure 1 (c)). The former is intuitive and could be achieved via Eq. 2, which naturally corresponds to these regions. However, the multi-modal structure of unknown-class data is underspecified, i.e., different unknown classes should not occupy the same region. By contrast, the latter is able to implicitly separate different unknowns and model the relationships between known and unknown classes. To make the model expandable, our key idea

𝑔

ℎ

Target

…

Memory bank

feature

logits

𝑁(𝑥)

Nearest neighbor

Estimated class conditionals

Prototype-based

classifier

Self-training

Semantic

consistency

𝑓

!

Source-trained

model (fixed)

𝑔

ℎ

Source

…

embedding

Logits

known

unknown

Eq. (1)

Eq. (2)

Eq. (3)

Eq. (4)

real known-class

virtual unknown-class

real known-class

real known-class

featurizer

classifier

Compaction

Disambiguation

is to compact the source embedding space by (i) making real known samples give response to both known and unknown classes, and (ii) inserting virtual unknown samples between known-class pairs.

Optimization on real known-class samples. In addition to the original source classes, we introduce a set of virtual classes C v within the source embedding space to mimic the existence of unknown classes. To accommodate the virtual unknown classes, we regularize the embedding space by pushing known-class samples closer to the decision boundaries:

̸

where f ˆ u ( x ) is the unknown's logit, having the largest activation among C v .

Optimization on virtual unknown-class samples. Despite the activation of the network's response to the introduced unknown classes, the 'winner-takes-all' nature of softmax-based classification still leaves the unknown category unable to compete effectively with known ones (cf. Figure 2). To mimic test environments, we synthesize virtual unknown samples. Technically, we synthesize unknown samples at the feature level for model regularization, without using external data, i.e., dashed points in Fig. 1 (c). We introduce manifold mixup [64] for synthesizing unknowns in the between-class regions, which are less confident for current decision boundaries. For two random samples x i and x j from different classes, we mix their embedding features as: ˆ z = µ · g ( x i ) + (1 -µ ) · g ( x j ) , where µ is the mixing coefficient. The optimization objective for the synthesized unknown ˆ z is defined as:

̸

where ˆ y represents the label of ˆ z regarding unknown class, i.e., having the largest activation among C v . h k ′ (ˆ z ) is the known's logit, having the largest activation among C s . The first term in Eq. 4 is a standard self-training loss. Similar to Eq. 3, the second term activates the response of ˆ z to its most related known class. In essence, apart from the standard classification loss (the first term in Eq. 3 and Eq. 4), we activate the response of the real known class towards unknowns (Eq. 3) and the response of the virtual unknown class towards the known ones (Eq. 4).

3.2 Test-Time Target Disambiguation

Although we have allocated the embedding space for unknown classes, how to deploy the sourcetrained model on real test data is yet to be thoroughly studied. In particular, we identify two major challenges: (i) optimality gap between source and target domains, and (ii) catastrophic forgetting in open and dynamic test environments. In OTDG, we define the optimality gap as follows.

Definition 1 (Optimality Gap) Let H ⊆ { h | h : Z ↦→ Y} be the hypothesis class. ε S ( · ) and ε T ( · ) denote the expected risk on source and target domains. For any hypothesis h , we have ε T ( h t ) < ε T ( h ∗ ) , where h ∗ = arg min h ∈H ε S ( h ) + ε T ( h ) and h t = arg min h ∈H ε T ( h ) .

Definition 1 suggests that it is not feasible to find a universal optimal classifier that applies to both source and target domains. In that sense, the classifier, initially trained on source data, needs additional refinement to adapt effectively to the target patterns. Technically, we resort to TTA [34] to mitigate the above issues using unlabeled test data in an online manner. Conventional training-based TTA methods [67] usually need batches of data for self-training ( e.g. entropy minimization) and/or heuristic self-supervision tasks [10]. On the other hand, training-free methods [23] require expensive tweaking of the threshold and only bring a marginal performance gain. Moreover, these approaches struggle to handle open-class scenarios, making them susceptible to negative transfer ( i.e. semantic misalignment). To address these challenges, our proposed target disambiguation stage aligns the unlabeled target samples to their corresponding class patterns through the following process.

We construct a memory bank S = { S 1 , · · · , S |C s | + |C v | } for memorizing the embedding z and logits f ( x ) (or h ( z ) ) of target samples. We compute a set of class prototypes { p k } |C s | + |C v | k =1 based on logits in S . Following [23, 24], the memory bank is initialized with the weights of the linear classifier. Let N ( x ) be the Nearest Neighbor (NN) of x in S . For each test sample x 3 , we search its NN in S by: N ( x ) := { z ∈ S | sim ( g ( x ) , z ) ≤ θ NN } , where sim ( · ) is the cosine similarity and θ NN is a threshold to control the number of NN. The model predictions would be given by the similarity between sample embedding and the class prototype, i.e., p ( y | x ) ∝ sim ⟨ p k , z ⟩ . Formally, for z ∈ N ( x ) , the likelihood of the prototype-based classifier assigning z to the k -th class can be calculated as follows:

where τ is a temperature scaling parameter. We estimate the class conditionals with N ( x ) as:

where ✶ ( · ) is an indicator function. Then, we can update the global class prototype computed from the whole S in a moving-average style, p k ← µ p k +(1 -µ )ˆ p k , (7)

where µ ∈ [0 , 1] is a preset scalar and ˆ p k can be regarded as the local class prototype. Given a batch of test samples B t , we use self-training for model updating ( g and h ), i.e., minimizing the cross-entropy loss between classifier's prediction f ( x ) and the estimated class prior distribution p k :

Semantic consistency. To resist catastrophic forgetting during the online adaptation process, we enforce semantic consistency between the output of f 0 (source-trained model) and f I (target model) by optimizing the cross-entropy loss between their predictions:

where I represents the number of iterations and f 0 is fixed throughout the testing phase.

Reliable sample selection. In the early stage of training, the estimation of target pseudo labels may be unreliable and thus is risky to error accumulation. To improve the quality of pseudo labels and

reduce the influence of false class estimations, we introduce an entropy-based weighting strategy to select reliable samples. Specifically, we define the scoring function as follows:

S ( x ) = ✶ ( H ( x ) < θ 0 ) exp( θ 0 -H ( x )) , (10) where H ( · ) is the Shannon entropy of sample and θ 0 is a pre-defined threshold. In this way, we can allocate larger weights to target samples with lower uncertainties and smaller weights to those with higher uncertainties, effectively prioritizing more confident predictions.

Test-time training objective. Formally, the overall optimization objective can be formulated as:

where λ is a trade-off parameter. The proposed L TTT embraces the complementary strengths of parametric (softmax-based) and non-parametric (prototype-based) classifiers.

4 Experiments

4.1 Setup

Benchmarks. We conduct extensive experiments on four standard DG benchmarks to verify the effectiveness of CODA. (1) PACS [28] has 9,991 images and presents remarkable distinctions in image styles. It is comprised of four domains each with seven classes, i.e., Photo , Art Painting , Cartoon , and Sketch . Dog , elephant , giraffe , and guitar are used as C s while the remaining 3 classes are C u t . (2) Office-Home [63] is gathered from both office and home environments, and its domain shifts originate from variations in viewpoint and image style. It has 15,500 images of 65 classes from four domains, i.e., Artistic , Clipart , Product , and Real World . Arranged in alphabetical order, the initial 15 classes are designated as C s , and the remaining 50 classes are assigned to C u t . (3) Office-31 [48] encompasses 31 classes harvested from three distinct domains: Amazon , DSLR , and Webcam . The 10 classes shared by Office-31 and Caltech-256 [18] are used as C s . In alphabetical order, the final 11 classes, combined with C s , constitute C t . (4) Digits , a dataset varying in background, style, and color, encompasses four domains of handwritten digits, including MNIST [26], MNISTM [17], SVHN [42], USPS [22], and SYN [17]. We utilize MNIST as the source domain, while the other datasets serve as target domains. Numbers from 0 to 4 make up C s .

Evaluation Protocols. In line with prior works [5, 90, 77], we utilize the H-score ( hs ) [16] as our main evaluation criterion. The hs score balances the significance between known and unknown classes, emphasizing that both groups should have high and equivalent accuracy. Additionally, the hs score circumvents the trivial solution where a model would predict all samples as known classes. The classification accuracy for both known ( acc k ) and unknown ( acc u ) classes are also reported.

Implementation Details. For PACS, Office-Home, and Office-31, we employ ResNet-18 [19], pre-trained on ImageNet, as the backbone network. For Digits, we employ the LeNet [25] with the architecture arranged as conv-pool-conv-pool-fc-fc-softmax . The training is performed using SGD with a momentum of 0.9 for 100 epochs, and we set the batch size to 64. Our experiments are built upon Dassl [87] (a PyTorch toolbox developed for DG), covering aspects of data preparation, model training, and model selection. We report the means over 5 runs with different random seeds.

4.2 Baselines

In experiments, we empirically compare CODA against five types of baselines: (1) OSDG: CM[90] and One Ring-S [77]. (2) OSDA: OSBP [49] and ROS [5]. (3) OD: MSP [20], LogitNorm [71], and DICE [56]. (4) SFDA: SHOT [35] and AaD [76]. (5) TTA: TTT [58], Tent [67], MEMO [80], TAST [24], and FAU [69]. Since TTA methods are incapable of directly handling unknown-class samples, we adopt the approach from [90], using the entropy of the softmax output as the final prediction. For ERM [60], we follow the same strategy for identifying unknowns.

4.3 Results

Our results are summarized in Table 2. For each dataset, CODA outperforms all compared methods by a considerable margin in terms of hs . For instance, in comparison to the previous best-performing OSDG baseline [77], CODA achieves increases in hs by 16.8% for PACS, 4.0% for Office-Home,

0

20

40

60

80

100

120

Epoch

10

20

30

40

50

60

70

80

Accuracy (%)

Acc_known

Acc_unknown

H-score

0

20

40

60

80

100

120

Epoch

10

20

30

40

50

60

70

80

Accuracy (%)

Acc_known

Acc_unknown

H-score

0

20

40

60

80

100

120

Epoch

10

20

30

40

50

60

70

80

Accuracy (%)

Acc_known

Acc_unknown

H-score

ERM (H-score)

(a) ERM [60]

(b) MSP [20]

(c) TAST [24]

(d) CODA (ours)

9.7% for Office-31, and 1.0% for Digits. If we focus on the hard generalization tasks, such as PACS, CODA exhibits larger performance gains than on other tasks. Moreover, three trends can be observed: (1) Compared to OSDA and SFDA methods that usually optimize with target data offline, CODA achieves superior performance in an online adaptation manner. (2) The acc k and acc u of Tent [67] and LogitNorm [71] exhibit significant imbalance as both methods tend to predict all data as known-class samples ( i.e. shortcut learning). This verifies the benefits of CODA in mitigating shortcut learning. (3) OD methods achieve very competitive results compared to other types of baselines. The rationale is that they usually do not involve test-time adjustment and thus have better stability. (4) The performance of different types of baseline methods varies across benchmarks. For instance, MEMO (TTA) achieves the second-best result in PACS but has relatively inferior performance in Office-Home and Digits. Instead, CODA exhibits consistent improvements on all benchmarks.

In addition, we provide performance comparisons of different methods ( i.e., ERM [60], MSP [20], TAST [24], and our CODA) as testing proceeds on the PACS dataset (trained on domain Art Painting ). To facilitate a fair comparison, both MSP and TAST will apply to the models that have been trained by our source compaction stage. Figure 3 shows that ERM and TAST substantially increase acc k and maintain it at a very high level, which severely impedes the improvements of hs . Interestingly, as the number of testing epochs increases, TAST underperforms

compared to ERM. By contrast, CODA dynamically harmonizes the relations between acc k and acc u ( i.e., suppresses acc k and hence allows acc u to grow), which is reflected by the monotonic increase of hs . Figure 4 shows Grad-CAM [52] visualizations of baseline (ERM) and our method (CODA) on the PACS dataset. We can see that the hot zones activated by CODA are more complete and reasonable, providing a reliable semantic understanding of the foreground object.

Battery

Bucket

Clipboard

Unknown

Computer

Bucket

Calculator

Battery

Clipboard

Unknown

w/o Disambiguation

Backpack

Unknown

Clipboard

Calendar

Computer

Unknown

Backpack

Clipboard

Computer

Bucket

w/ Disambiguation

w/o Disambiguation

w/ Disambiguation

4.4 Ablation Studies

Ablations of key components in CODA. We carry out ablation studies in Table 3, evaluating the effect of source compaction (SC) and target disambiguation (TD) proposed in CODA. When we exclude SC from CODA, model predictions are made based on the entropy of the output in conjunction with a predetermined threshold [90]. From the table, we can observe that adding SC and TD could improve the generalization per-

formance, which verifies their individual contributions for solving domain shifts and open classes. Moreover, our method that integrates both SC and TD achieves the best performance, revealing the synergistic effect between the two components.

Analysis on target disambiguation. In Figure 5, we plot the predictions on known- and unknownclass test samples from the models trained with and without target disambiguation. The left image is a battery (known), and the right image is a clipboard (unknown). For the known classes, a model lacking disambiguation often makes uncertain predictions, especially for hard samples that bear high resemblance to other classes. For the unknown class, a model without disambiguation tends to give a high response to an incorrect class and suppress responses to other classes. In contrast, our target disambiguation stage can achieve more accurate predictions by recovering the semantic relationships among classes from unlabeled data, thereby enhancing the model's generalization performance under both domain shift and open classes. In Figure 6 (a)-(b), we investigate the the impact of varying the test batch size on three methods, TAST [24], FAU [69], and CODA (ours). As the batch size varies, CODA consistently delivers superior performance compared to TAST and FAU, revealing the advantages of the proposed online adaptation strategy.

Analysis on unknown-aware training objective. We empirically compare different unknown-aware training objectives discussed in Section 2 and Section 3.1, i.e., Eq.(1)-(4) and their combinations. For a fair comparison, we do not involve any TTA strategies including our target disambiguation. The results are reported in Table 4. We can observe

that our full source compaction is clearly better than its variants, revealing the superiority of our optimization procedures on both real known-class samples and virtual unknown-class samples.

Analysis on unknown classes. In Figure 6 (c)-(d), we study the impact of varying the number of known classes on three methods, ERM [60], One Ring-S [77], and CODA (ours). Note that the total number of classes ( i.e. |C s ∪ C t | ) remains unchanged. Even when the number of known classes is small, CODA still exhibits superior performance. This advantage remains consistent as the number of known classes changes. Consequently, CODA is capable of handling extreme scenarios.

Feature visualization. We use t -SNE [59] to visualize the features of four models on Office-31 dataset, i.e., ERM, One Ring-S, Source Compaction, and CODA (full). The results are depicted in Figure 7, where various colors, excluding gray, signify different known classes, and gray points represent all unknown classes. It is noteworthy that the embedding features learned by two baselines (ERM and One Ring-S) fail to present a clear separation, resulting in ambiguous boundaries among

0

30

60

90

120

150

180

210

240

Batch size

20

30

40

50

60

70

H-score (%)

Ours

FAU

TAST

6

9

12

15

18

21

24

Known classes

20

30

40

50

60

70

H-score (%)

Ours

ERM

One Ring-S

2

3

4

5

6

Known classes

20

30

40

50

60

70

H-score (%)

Ours

ERM

One Ring-S

(a)

(b)

(c)

(d)

(a) ERM

(b) One Ring-S [77]

(c) Source Compaction

(d) CODA (full)

different classes particularly between known and unknown ones. Instead, CODA can learn the intrinsic structure ('manifold') of the target samples, providing more discriminable clustering patterns.

5 Related Work

Domain Generalization (DG). DG is concerned with the training of a model using data from either multiple or a single source domain, with the goal of achieving good generalization to previously unseen target domains. Mainstream approaches usually involve invariant learning and robust learning with elaborate training objectives. Based on the focus of this paper, we classify existing methods into three categories and provide descriptions as follows. (1) Closed-set DG. Existing methods can be roughly divided into four categories: feature matching-based [32, 31, 40, 91, 83], decompositionbased [46, 45, 13, 39, 54, 36, 78], augmentation-based [65, 85, 86, 74, 41, 88, 8], and meta-learningbased approaches [29, 33, 30, 37, 7]. (2) Open-set DG. It is worth noting that very few works have delved into the exploration of DG in open-set scenarios. A handful of recent studies [53, 90, 77, 9] started to consider the existence of both known and unknown classes in novel DG settings, such as Open-Set DG (OSDG) [90, 77]. For example, Yang et al. [77] hold the view that any category other than the ground-truth can be considered as part of the unknown classes. Chen et al. [9] first activate the unknown's logit via an unknown-aware training loss and then introduce a test-time adjustment strategy to refine the model prediction. Zhu et al. [90] rely on heuristic thresholding mechanisms to distinguish known- and unknown-class samples. (3) DG by test-time adaptation. According to Dubey et al. [14], a model trained solely on source data will inevitably experience an 'adaptivity gap' when it is directly employed on target domains, emphasizing the necessity of on-target adaptation. Grounded on this insight, several recent works [79, 73, 72, 11] resort to TTA for mitigating the adaptivity gap, such as adaptive risk minimization [79], energy-based sample adaptation [72], and improved self-supervised learning tasks [11].

Test-Time Adaptation (TTA). TTA [34] is an emerging paradigm that has demonstrated immense potential in adapting pre-trained models to unlabeled data during the testing phase. A plethora of approaches [35, 58, 67, 23, 43, 10, 80, 69] have been developed to improve the predictions of sourcetrained models on target domain with online training/adaptation strategies. TTT [58] introduces self-supervised learning tasks ( e.g. rotation classification) to both source and target domains. Tent [67] leverages the batch statistics of the target domain and optimizes the channel-wise affine parameters by entropy minimization. T3A [23] proposes to use class prototypes for adjusting predictions and introduces a support set to memorize representative and reliable samples. TAST [24] improves T3A by proposing a nearest neighbor information induced self-training framework.

Out-of-Distribution Detection (OD). OD [75], which seeks to identify novel instances that the model has not encountered during training, is close to OTDG setting. Prevailing OD methods center on creating OOD scoring functions, for example, confidence-based techniques [4, 20, 21], distancebased scores [27, 51, 57], and energy-based scores [38, 55]. Although promising, the OD approach is limited to binary classification problems and lacks the capability to effectively explore domain shift and open class challenges in the test data.

There are also other topics related to open-world machine learning [44] that bear certain relevance to our work, including open-set recognition [50, 62], novel class discovery [15, 61], zero-shot learning [47, 70], and class-incremental learning [6, 82], to name a few. Compared to previous methods, our work addresses two types of open-world situations ( i.e., domain shift and open class), supporting generalization capabilities consistently throughout the training and inference phases.

6 Conclusion

In this paper, we solve the problem of OTDG where both domain shift and open classes may concurrently arise on the unseen test data. We introduce a two-stage framework (CODA) for efficiently learning what we don't know in the wild. At the training stage, we compact the embedding of source known classes and thus reserve space for target unknown classes. In the testing phase, we introduce a training objective to mitigate the optimality gap between domains while avoiding catastrophic forgetting. Empirically, CODA achieves superior performance on standard DG benchmarks.

References

Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In ICML , pages 145-155, 2020.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Shai Ben-David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In International Conference on Artificial Intelligence and Statistics , pages 129-136, 2010.

Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In CVPR , pages 1563-1572, 2016.

Silvia Bucci, Mohammad Reza Loghmani, and Tatiana Tommasi. On the effectiveness of image rotation for open set domain adaptation. In ECCV , 2020.

Francisco M Castro, Manuel J Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In ECCV , 2018.

Chaoqi Chen, Jiongcheng Li, Xiaoguang Han, Xiaoqing Liu, and Yizhou Yu. Compound domain generalization via meta-knowledge encoding. In CVPR , pages 7119-7129, 2022.

Chaoqi Chen, Luyao Tang, Feng Liu, Gangming Zhao, Yue Huang, and Yizhou Yu. Mix and reason: Reasoning over semantic topology with data mixing for domain generalization. NeurIPS , 35:33302-33315, 2022.

Chaoqi Chen, Luyao Tang, Leitian Tao, Hong-Yu Zhou, Yue Huang, Xiaoguang Han, and Yizhou Yu. Activate and reject: Towards safe domain generalization under category shift. In ICCV , pages 11552-11563, 2023.

Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR , pages 295-305, 2022.

Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In CVPR , 2023.

Noam Chomsky. Aspects of the Theory of Syntax , volume 11. MIT press, 2014.

Rune Christiansen, Niklas Pfister, Martin Emil Jakobsen, Nicola Gnecco, and Jonas Peters. A causal framework for distribution generalization. IEEE TPAMI , 2021.

Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for real-world domain generalization. In CVPR , 2021.

Enrico Fini, Enver Sangineto, Stéphane Lathuiliere, Zhun Zhong, Moin Nabi, and Elisa Ricci. A unified objective for novel class discovery. In ICCV , 2021.

Bo Fu, Zhangjie Cao, Mingsheng Long, and Jianmin Wang. Learning to detect open classes for universal domain adaptation. In ECCV , pages 567-583. Springer, 2020.

Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML , pages 1180-1189, 2015.

Boqing Gong, Yuan Shi, Fei Sha, and Kristen Grauman. Geodesic flow kernel for unsupervised domain adaptation. In CVPR , pages 2066-2073. IEEE, 2012.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR , pages 770-778, 2016.

Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR , 2017.

Rui Huang and Yixuan Li. Towards scaling out-of-distribution detection for large semantic space. In CVPR , 2021.

Jonathan J. Hull. A database for handwritten text recognition research. TPAMI , 16(5):550-554, 1994.

Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. NeurIPS , 34:2427-2440, 2021.

Minguk Jang and Sae-Young Chung. Test-time adaptation via self-training with nearest neighbor information. In ICLR , 2023.

Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation , 1(4):541-551, 1989.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE , 86(11):2278-2324, 1998.

Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS , pages 7167-7177, 2018.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV , pages 5542-5550, 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI , 2018.

Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV , pages 1446-1455, 2019.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR , pages 5400-5409, 2018.

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV , pages 624-639, 2018.

Yiying Li, Yongxin Yang, Wei Zhou, and Timothy Hospedales. Feature-critic networks for heterogeneous domain generalization. In ICML , pages 3915-3924, 2019.

Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361 , 2023.

Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML , pages 6028-6039. PMLR, 2020.

Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie-Yan Liu. Learning causal semantic representation for out-of-distribution prediction. NeurIPS , 34, 2021.

Quande Liu, Qi Dou, and Pheng-Ann Heng. Shape-aware meta-learning for generalizing prostate mri segmentation to unseen domains. In MICCAI , pages 475-485. Springer, 2020.

Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. In NeurIPS , 2020.

Divyat Mahajan, Shruti Tople, and Amit Sharma. Domain generalization using causal matching. In ICML , pages 7313-7324, 2021.

Toshihiko Matsuura and Tatsuya Harada. Domain generalization using a mixture of multiple latent domains. In AAAI , 2020.

Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR , pages 8690-8699, 2021.

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.

Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh Ap. Generalization on unseen domains via inference-time label-preserving target projections. In CVPR , pages 12924-12933, 2021.

Jitendra Parmar, Satyendra Chouhan, Vaskar Raychoudhury, and Santosh Rathore. Open-world machine learning: applications, challenges, and opportunities. ACM Computing Surveys , 55(10):1-37, 2023.

Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via commonspecific low-rank decomposition. In ICML , pages 7728-7738, 2020.

Mateo Rojas-Carulla, Bernhard Schölkopf, Richard Turner, and Jonas Peters. Invariant models for causal transfer learning. JMLR , 19(1):1309-1342, 2018.

Bernardino Romera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. In ICML , 2015.

Kate Saenko, Brian Kulis, Mario Fritz, and Trevor Darrell. Adapting visual category models to new domains. In ECCV , pages 213-226. Springer, 2010.

Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In ECCV , pages 153-168, 2018.

Walter J Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. IEEE TPAMI , 35, 2012.

Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In ICLR , 2021.

Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In ICCV , pages 618-626, 2017.

Yang Shu, Zhangjie Cao, Chenyu Wang, Jianmin Wang, and Mingsheng Long. Open domain generalization with domain-augmented meta-learning. In CVPR , pages 9624-9633, 2021.

Xinwei Sun, Botong Wu, Xiangyu Zheng, Chang Liu, Wei Chen, Tao Qin, and Tie-Yan Liu. Recovering latent causal factor for generalization to distributional shifts. NeurIPS , 34, 2021.

Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. In NeurIPS , 2021.

Yiyou Sun and Yixuan Li. Dice: Leveraging sparsification for out-of-distribution detection. In ECCV , 2022.

Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. In ICML , 2022.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML , pages 9229-9248. PMLR, 2020.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR , 9(11), 2008.

Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks , 10(5):988-999, 1999.

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Generalized category discovery. In CVPR , 2022.

Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In ICLR , 2022.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , pages 5018-5027, 2017.

Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Ioannis Mitliagkas, David Lopez-Paz, and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In ICML , 2019.

Riccardo Volpi and Vittorio Murino. Addressing model vulnerability to distributional shifts over image transformation sets. In ICCV , pages 7980-7989, 2019.

Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS , pages 5334-5344, 2018.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR , 2021.

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. TKDE , 2022.

Shuai Wang, Daoan Zhang, Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In CVPR , 2023.

Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST) , 10(2):1-37, 2019.

Hongxin Wei, Renchunzi Xie, Hao Cheng, Lei Feng, Bo An, and Yixuan Li. Mitigating neural network overconfidence with logit normalization. In ICML , 2022.

Zehao Xiao, Xiantong Zhen, Shengcai Liao, and Cees GM Snoek. Energy-based test sample adaptation for domain generalization. In ICLR , 2023.

Zehao Xiao, Xiantong Zhen, Ling Shao, and Cees GM Snoek. Learning to generalize across domains on single test samples. In ICLR , 2022.

Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, and Marc Niethammer. Robust and generalizable visual representation learning via random convolutions. In ICLR , 2021.

Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-distribution detection: A survey. arXiv preprint arXiv:2110.11334 , 2021.

Shiqi Yang, Yaxing Wang, Kai Wang, Shangling Jui, et al. Attracting and dispersing: A simple approach for source-free domain adaptation. In NeurIPS , 2022.

Shiqi Yang, Yaxing Wang, Kai Wang, Shangling Jui, and Joost van de Weijer. One ring to bring them all: Towards open-set recognition under domain shift. arXiv preprint arXiv:2206.03600 , 2022.

Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain generalization. In CVPR , pages 8024-8034, 2022.

Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. NeurIPS , 34:23664-23678, 2021.

Marvin Mengxin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In NeurIPS , 2022.

Long Zhao, Ting Liu, Xi Peng, and Dimitris Metaxas. Maximum-entropy adversarial data augmentation for improved generalization and robustness. NeurIPS , 33:14435-14447, 2020.

Da-Wei Zhou, Fu-Yun Wang, Han-Jia Ye, Liang Ma, Shiliang Pu, and De-Chuan Zhan. Forward compatible few-shot class-incremental learning. In CVPR , 2022.

Hong-Yu Zhou, Yizhou Yu, Chengdi Wang, Shu Zhang, Yuanxu Gao, Jia Pan, Jun Shao, Guangming Lu, Kang Zhang, and Weimin Li. A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics. Nature Biomedical Engineering , 7:743-755, 2023.

Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. TPAMI , 2022.

Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In AAAI , pages 13025-13032, 2020.

Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV , pages 561-578, 2020.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain adaptive ensemble learning. TIP , 30:8008-8018, 2021.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR , 2021.

Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In ICML , pages 27222-27244, 2022.

Ronghang Zhu and Sheng Li. Crossmatch: Cross-classifier consistency regularization for open-set single domain generalization. In ICLR , 2022.

Wei Zhu, Le Lu, Jing Xiao, Mei Han, Jiebo Luo, and Adam P Harrison. Localized adversarial domain generalization. In CVPR , pages 7108-7118, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a two-stage framework for the task of open-class domain generalization. During training, it introduces virtual unknown classes for the learning objective to insert unknowns into the latent space by compacting the embedding space of known classes. During the test phase, it proposes a prototype-based test-time adaptation pipeline, which involves (1) consistency between the predicted class distributions and the estimated class conditionals; (2) class-wise sample reliability for ensuring the quality of target pseudo labels; (3) semantic consistency between the source-trained model and target model, to disambiguate the decision boundaries between known and unknown classes. Experimental results on different DG benchmarks demonstrate its effectiveness.\n\nSTRENGTHS:\n1. The paper studies a practical problem that has been rarely explored, open test-time domain generalization, which considers the co-occurrence of domain shift and open class. The comparison with other related settings is clearly stated and provides good intuitions.\n\n2. The proposed framework, CODA, is novel and technically sound. The introduced compaction and disambiguation process is easy to implement, and the computational overhead is relatively small. Overall, the paper is well organized, and the advantage of CODA is clearly demonstrated.\n\n3. The experiments confirm the effectiveness of the method. In particular, different types of baselines, such as DG, DA, TTA, and OOD detection, are explicitly compared. Ablations and visualization further justify the contributions of individual components.\n\nWEAKNESSES:\n1. There are some inconsistencies that need to be clarified. Firstly, the authors initially utilize the term 'adaptivity gap', but introduce a new terminology, 'optimality gap', in Definition 1. What are their distinction and correlation? Secondly, the proposed setting called OTDG appears to underscore the process of test-time operation. However, I found that the source training stage offers large improvements according to the reported results. As such, I question whether the use of 'test-time domain generalization' is fitting. Thirdly, in Figure 2, the authors use dashed points to signify virtual unknown samples. However, these samples are produced by manifold mixup at the feature level, which makes them distinct from the original data.\n\n2. Although Figure 1 provides an intuitive comparison among different learning objectives, I struggle to fully comprehend why Equation (3) is superior to Equation (2). The advantages of employing multiple virtual prototypes versus a single virtual prototype warrant further discussion and comparison. \n\n3. Basically, the target disambiguation is a prototype-based classification process. As the target distribution contains some open classes, how the disambiguation is done by only using the source-learned virtual prototypes? Moreover, could the proposed CODA be extended to a more challenging scenario that the open classes are further classified into different clusters?\n\n4. How are the experiments conducted with different types of baselines, including aspects like train-validation split and model selection? Further clarification and details regarding the fairness of comparison should be provided.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nLimitations are discussed in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research paper introduces a novel experimental configuration referred to as Open Test-time Domain Generalization (OTDG), which can be viewed as an amalgamation of domain generalization, test-time adaptation, and open-set recognition. In order to address this particular experimental configuration, the authors put forward a fresh conceptual framework termed Compaction and Disambiguation (CODA). The fundamental concept behind this framework entails the imposition of restrictions on decision boundaries during the training phase, leveraging labeled source data, with the aim of ensuring an ample latent space to accommodate future classes. Subsequently, these decision boundaries are further refined during the test phase through the utilization of unlabeled target data.\n\nSTRENGTHS:\n1, The experimental setting introduced in this study represents a novel paradigm. Although its practical viability is yet to be established, it presents a unique problem that has not been previously encountered.\n\n2, The paper effectively presents a well-structured argument, providing a reasonable proposition of loss functions aimed at addressing the OTDG problem.\n\n3, The experiments conducted in this study are meticulously executed and accompanied by illustrative figures that effectively convey the findings. The evaluation involves a comprehensive comparison with recent state-of-the-art works that are closely related to the subject matter, ensuring a robust analysis of the proposed approach.\n\nWEAKNESSES:\nThe motivation behind the concept of Open Test-time Domain Generalization (OTDG) in this study may benefit from further clarification. It would be valuable to present realistic scenarios that exemplify the challenges addressed by this setup. By providing concrete examples or case studies that demonstrate the practical relevance of OTDG, the study can establish a stronger connection between the proposed approach and its potential real-world applications. This would enhance the understanding of how OTDG can be applied in practical scenarios and highlight its significance in addressing specific challenges faced in domains that require generalization, test-time adaptation, and open-set recognition. The proposed solution in this study demonstrates a reasonable level of novelty in addressing the challenges posed by the specific OTDG problem. While it is acknowledged that there may be similarities with previous related works, this is considered a minor aspect. The study's main focus lies in presenting a solution that effectively tackles the challenges unique to the OTDG problem, highlighting the authors' contribution in the field. By emphasizing the novel aspects of the proposed solution and its potential impact, the study provides valuable insights and advancements within the context of the OTDG problem.\n\nMinor weakness: no codes in the supplementary material.\n\nQUESTIONS:\nAs I said above, can the authors list some practical scenario of the OTDG setup and convince me this is not a hand made setup just for research.\n\nLIMITATIONS:\nOne of the main limitations of this study revolves around the practicality of the proposed setup. It is natural to question whether the setup accurately reflects real-world scenarios and if it can be applied in practical contexts. To gain more insight into the effectiveness and applicability of the Open Test-time Domain Generalization (OTDG) problem, it would be valuable to explore realistic scenarios that closely resemble the challenges faced in practical settings. This would help establish a stronger connection between the proposed approach and its potential real-world applications.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the problem of domain adaptation. In particular, in addition to the already-studied setting of domain shift at test time, this paper considers the practical scenario of unknown classes being presented at test time. Often current models will incorrectly predict a known class label for examples from unknown classes. This paper addresses this by: Compaction and Disambiguation. Compaction forces the known classes embeddings to be more compacted, with regions in between classes being reserved for unknown classes through the introduction of virtual unknown classes. Disambiguation, done at test time, is for test time adaptation. The objective in this phase optimizes 1) consistency between model output predictions using a nearest neighbor memory bank of class prototypes 2) consistency between source and target model to avoid catastrophic forgetting.\n\nSTRENGTHS:\n1) Introduction of a novel problem: unknown classes at test time in addition to domain shift \n\n2) Thorough ablation on various components of methods to illustrate why every choice is necessary\n\n3) Strong empirical results (SOTA across several datasets - I am not too familiar with what the standard datasets for domain generalization are though)\n\nWEAKNESSES:\n1) More background on test-time adaptation can be given before introducing the \"disambiguation\" protocol since many readers (like myself) may not be very familiar with this\n\nQUESTIONS:\nCan the authors explain in more detail why the known-class examples must activate unknown-class logits?\n\nLIMITATIONS:\nThe authors could include a study on how the test time adaptation affects source domain accuracy i.e. what is it before and after the disambiguation protocol? I'm not sure if these results have been included or discussed thoroughly. Since the TTA loss has a term to encourage consistency with source model, how good the model is at this objective should be evaluated.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel problem formulation termed Open Test-Time Domain Generalisation, which accounts for the presence of both domain shift and unseen classes during the testing phase. To tackle this problem, a two-stage methodology is proposed, comprising Compaction and Disambiguation. The primary objective of the Compaction stage is to facilitate the creation of a region in the embedding space that can accommodate the incoming unknown classes at test-time. This is accomplished by introducing synthetic unknown classes into the latent space by compacting the embedding space associated with the known classes from the source domain.  At training time, in lieu of unknown class data, manifold mixup is used to synthesise the unknown class training data and a classifier is trained that has additional decision boundaries corresponding to the unknown classes.\n\nWhile the compaction stage is proposed during training to accommodate new classes, the disambiguation phase is proposed to handle the potential domain shift in the target domain. For the disambiguation phase, authors adopt existing TAST [ref 20 from paper] methodology and apply it to a setting with unknown classes.\n\nExperimental results show that  the current method has been exhaustively tested against baselines.\n\nSTRENGTHS:\n+ Novel Problem Formulation: The paper introduces a novel problem formulation called Open Test-Time Domain Generalization, which addresses the challenges of domain shift and unseen classes during the testing phase. This formulation extends the scope of traditional domain generalization methods and accounts for realistic scenarios where new classes can emerge at test time.\n+ Integration of Manifold Mixup: To address the lack of unknown class data during training, the paper leverages manifold mixup to synthesize the unknown class training data. This integration allows the model to learn decision boundaries corresponding to the unknown classes. By utilizing this data augmentation technique, the proposed method enhances the model's ability to generalize to unknown classes and improve overall performance.\n+ Extensive Experimental Evaluation: The paper demonstrates the thoroughness of the research by subjecting the proposed method to exhaustive testing against relevant baselines.\n\nWEAKNESSES:\nThe work brings together ideas from different existing methods and provides a novel way to tackle the problem setting of Open Test-Time Domain Generalisation. While there are a certain amount of novel claims made in the paper, they are often unsubstantiated. We have provided some pointers to where further explanations are required. Additionally, we recommend that paper be thoroughly proofread.\n\n* The paper lacked a sufficient level of explicitness in conveying the necessary information. Important details or background knowledge seemed to be assumed or left to inference, which makes it challenging for readers to follow the logical flow of the study. It would greatly benefit the clarity of this work if additional context, explanations, and descriptions could be provided to ensure a more straightforward understanding for readers who may not possess the same background knowledge or expertise as the authors. For instance, (a)\nWhile Eq.5 shows that the prototype-based classifier assigns 'z' to one of |C_s|+N classes, line 60 of the supplementary file contradicts that by assuming all N classes to be one class. Please clarify. (b) The mechanics of support set, its constitution and updation are ill-defined.\n* While the paper asserts the mitigation of catastrophic forgetting in line 16, it lacks empirical evidence to validate this claim. Therefore, it is recommended to include relevant results in order to substantiate the effectiveness of the proposed approach in addressing catastrophic forgetting. Reporting ablation study of semantic consistency loss on source domain known classes is advised.\n* In line 133, the inequality seems to be incorrect and the assertion in lines 134-135 does not trivially follow from the inequality. \n* Lines 132 through 136 introduce the notion of optimality gap which we believe is a misnomer as this is defined as “adaptivity gap” [ref 11 from paper]. However, regardless of the appropriateness of the term optimality gap, we believe that the problem that TTA resolves is domain shift, rendering the previous lines irrelevant.\n\nQUESTIONS:\n* While the statement “...we activate the response of the real known class towards unknowns (Eq. 3)” from line 124 is adequately explained, we believe that further explanation of line 125, “... and the response of the virtual unknown class towards the known ones” is required.\n* In line 111, y_hat shouldn’t be the index but rather a one-hot vector with non-zero entry at the index that y_hat currently holds. Similarly with line 121, “ y’ ” is incorrectly defined.\n* In Eq.3 and Eq.4, a new undefined term ‘C’ is introduced.\n* We request the authors to use notation consistent with TAST [ref 20 from paper] when building upon their contributions, for increased accessibility.\n* Please justify the choice of scoring function in line 165.\n* In line 185, the unknown class set cannot include the source domain class set.\n* Line 205 requires clarification.\n* While Eq.5 shows that the prototype-based classifier assigns 'z' to one of |C_s|+N classes, line 60 of the supplementary file contradicts that by assuming all N classes to be one class. Please clarify.\n\nLIMITATIONS:\nNone in particular", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}