Reconstruct and Match: Out-of-Distribution Robustness via Topological Homogeneity
Authors: Chaoqi Chen
Luyao Tang
Hui Huang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Out-of-Distribution
Domain Generalization
Test-Time Adaptation
Topological Homogeneity
Primary area: safety_in_machine_learning

Abstract: Since deep learning models are usually deployed in non-stationary environments, it is imperative to improve their robustness to out-of-distribution (OOD) data. A common approach to mitigate distribution shift is to regularize internal representations or predictors learned from in-distribution (ID) data to be domain invariant. Past studies have primarily learned pairwise invariances, ignoring the intrinsic structure and high-order dependencies of the data. Unlike machines, human recognizes objects by first dividing them into major components and then identifying the topological relation of these components. Motivated by this, we propose Reconstruct and Match (REMA), a general learning framework for object recognition tasks to endow deep models with the capability of capturing the topological homogeneity of objects without human prior knowledge or fine-grained annotations. To identify major components from objects, REMA introduces a selective slot-based reconstruction module to dynamically map dense pixels into a sparse and discrete set of slot vectors in an unsupervised manner. Then, to model high-order dependencies among these components, we propose a hypergraph-based relational reasoning module that models the intricate relations of nodes (slots) with structural constraints. Experiments on standard benchmarks show that REMA outperforms state-of-the-art methods in OOD generalization and test-time adaptation settings.

Reconstruct and Match: Out-of-Distribution Robustness via Topological Homogeneity
Chaoqi Chen'
Luyao Tang² Hui Huang
Coolleee of Computer Science and Software Engineering, Shenzhen University
School of Informatics, Xiamen University
cgchen19940gaiil.com lytangdatu. edu. hhzhiyan@gmail.com com
Abstract
Since deep learning models are usually deployed in non-stationary environments, it is imperative to improve their robustness to autto--distribution (OOD) data. A com- mon approach to mitigate distribution shift is to regularize internal representations or predictors learned from in-distribution (ID) data to be domain invariant. Past studies have primarily learned pairwise invariances, ignoring the intrinsic structure and high-order dependencies of the data. Unlike machines, humans recognize objects by first dividing them into major components and then identifying the topological relation of these components. Motivated by this, we propose Recon- struct and Match (REMA). general learning framework for object recognition tasks to endow deep models with the capability of capturing the topological ho- mogeneity of objects without human prior knowledge fine-grained annotations. To identify major components from objects, REMA introduces a selective slot- based reconstruction module to dynamically map dense pixels into sparse and discrete set of slot vectors an unsupervised manner. Then, to model high-order relational
dependencies among these components, propose hypergraph-berr reasoning module that models the intricate relations nodes (slots) with structural constraints. Experiments standard benchmarks show that REMA outperforms stateoof-the-ar methods in OOD generalization and test-time adaptation settings.
Introduction
Distribution shift has emerged as a major challenge for the success of machine learning systems [34, 25). Although deep learning models are believed to generalize well to in-distribution (ID) data, well-trained model deployed in the open world often encounters out-of-distributtin (OOD) data whose contexts may differ from the training distribution, resulting in dramatic performance degeneration and raising concerns about model safcty in many high-staking applications, such as autonomous driving and medical diagnosis. This gives rise to the importance of OOD generalization [92, 76], which aims to build a robust learning machine that can perform well in unseen test environments.
Regarding OOD generalization, central theme is how to learn general features from training data that can be extrapolated test distributions. Following this idea, a plethora of OOD generalization methods have been proposed over the past few years, including domain alignment [42], latent feature disentanglement [61, 48, 85), meta-learning [40, 41J, invariant risk minimization [2, 1, 96], and augmentation-guinn invariant predictor [74, 83, 95], to name a few. On the other hand, given the natural adaptivity gap between training and test distributions [15], recent works [29, 30, 81, 11, 8] attempt to further enhance the source-trained model through test-time adaptation [46], which leverages unlabeled target samples to update the model in an online manner. These two series studies address distribution shifts during training and inference and have been proven to work synergistically [8].
"Conesponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Pairwise Alignment
Pairwise Alignment
Relational Modeling
Dense & Continuous
Sparse & Discrete
Disentanglement
Reconstruction
(a) Idea
Popular approach
(c) Our framework
Figure 1: Motivation of the proposed REMA. (a) Ideally, by partitioning the latent factors into common and specific parts, aligning the common factors directly enables perfect matching of two different distributions. (b) Since the latent features learned by deep networks are typically dense and continuous, well-perfomming methods seek direct alignment without exploring the inherent structures. (c) Our REMA introduces sparse and discrete element - slot [51] to serve as bridge for both reconstruction and relational modeling, avoiding explicit disentanglement in the latent space.
Albeit their general efficacy for various tasks, these prior efforts largely overlook the topological structure of the image data. As consequence, existing OOD generalization methods highly rely specialized regularization objectives and are significantly less interpretable than human vision systems. Unlike machines, human object recognition [4] typically involves initially decomposing objects into several major components (e.g., keypoints), followed by identifying the structural relationships among these components, and finally making predictions by comprehensively considering the main components and their inherent relations. To this end, a critical question remains open in the field:
How devise a unified framework that imitates the human vision process for OOD generalization?
In this work, we believe that OOD generalization requires consideration of two key aspects: 1) how to represent the structure data, and 2) how to model the relationships between different entities.
Grounded on these insights, we propose Reconstruct and Match (REMA), a general framework to endow deep models with the capability of capturing the topological homogeneity of objects without using human prior knowledge or fine-grained attribute annotations. Figure illustrates the motivation. To identify major components from objects, REMA introduces slot-based reconstruction module to map dense pixels into sparse set of slot vectors in an unsupervised manner. This module encourages the deep model to reduce unwanted redundancy and preserve those predominant parts (the words "component" and "part" are used interchangeably). Then, to model and reason the high- order dependencies among these components, we propose hypergraph--asser matching module that discovers the relations of slots with structural constraints, i.e., topological homogeneity of the object. The main contributions this paper are summarized as follows:
We propose REMA, novel OOD generalization framework mitigate distribution shifts in deep learning models by imitating the human visual recognition process.
We introduce a self-supervised reconstruction process to identify informative parts from objects without additional supervision, and hypergraph matching module to reason the high-order part-based object relationships and interactions across domains.
Experiments on six widely used benchmarks demonstrate that REMA outperforms state-of- the-art methods in OOD generalization and test-time adaptation settings.
2 Preliminaries
Problem Setup. Assume that X' is the input space, 2 is the latent space, and ) is the output space. The predictor f = o g is comprised of a featurizer g : X' ~ 2 that learns to extract embedding features, and classifier h Z + y that makes predictions based on the extracted features. The goal

---

Selective Slot-based Reconstruction (S5R)
High-Order Relational Reasoning (HORR)
Hyporgraph Matching
Encoder
Sinkhorn Layer
Optmmum
AssR
Extracted
Slot Selection
Reconstructed
ViuaiitoSraphh
Graph Node
Hypersdge
Features
Figure 2: Overview of the proposed REMA, which consists of two key modules, ie., SSR and HORR. (1) Abstraction: Slot-based reconstruction to discover the main components from the data by binding objects with set of discrete vectors; (2) Reasoning: Introduce high-order relational inductive bias (i.e., topological homogeneity) to the network via the process of hypergraph construction, learning, and matching. HGNN means hypergraph neural networks.
of OOD generalization is to find predictor f X + y that generalizes well to all unseen target domains. In deep neural networks, empirical risk minimization (ERM) [72] is capable learning highly predictive features, making it the simplest baseline for this problem.
ERM Objective. We train the model by minimizing the empirical cross-entropy loss function:
arg min (= [((f((z), x)], DEG
where ptr is the training distribution, θ denotes the parameter space, and is the loss function. The trained model will be evaluated on a test set from a test distribution pts, where ptr + pts.
3 Proposed Method
In this section, provide detailed description of our Reconstruct and Match (REMA) framework. As shown in Figure 2, REMA consists two key modules tackling the main component discovery (Section 3.1) and high-order relational modeling and reasoning (Section 3.2).
3.1 On Discovering Main Components
For OOD generalization/aaapattioo Slomain-agnostion and non non-transferable (domain-related) information in the data. An ideal solution is disentangle latent representations an unsupervised manner (see Fig. 1(a),, which has been proven to be prohibitively difficult and even infeasible [50]. Therefore, most prior works (see Fig. ((b) aim learn Nomain-invariinn representations through elaborate feature alignment modules, such as adversarial training [21] and pseudo-labeling [37], without exploring the intrinsic composition of the features themselves. Despite their general efficacy, these methods may introduce significant amount of redundant and noisy information during the alignment process. For humans, when identifying an object, we tend to first look for its main components, which can be regarded as a process of information compression from learning perspective. Inspired by this, aim mimic the very first step of the human recognition process by identifying the main components from object images without human prior knowledge or fine-grained annotations.
issue is how to distinguish between transferable
To be specific, we introduce supervised reconstruction module based on Slot Attention [51], termed Selective based Reconstruction (SSR), to assist deep models in learning set of bast- representations for sparsely characterizing target object. Formally, given an input x, extracts set of feature embeddings E R xd, where N is the number of embedding and d, denotes embedding dimension. The Slot Attention module will first take set of slots E Kxx (K is the number of slots) and the feature map z, then project them to dimension de by lincar transformation fx for slots

---

and fgr fo for z. The Slot Attention will be trained as follows,
AopdateAA,v) v Ao attn(q, k)i
Ma
kq
attn - Ei=1 e
M
(2)
Σ attn(q, k)1;
vd,
( RKxd, k -f((z) E BR and [u(z) E RNxd, denote the query, key and value vectors respectively. A E RRNK stands for the attention matrix. Unlike self-attention, the queries in slot attention are a function of the slots ~ (s: fa, o)) E _Kxd,, and will be updated iteratively over I' iterations. The slots are initialized by random sampling from standard Gaussian distribution. Queries at iteration & are represented by q' = fs"s"), and the slot updating process is: s == update tn Q", k). v). After each iteration, a Gated Recurrent Unit (GRU) is employed on the slot representations s/+1 update their states, integrating new information while retaining relevant context from previous iterations.
where q fz)) E
Since the number of slots K is manually predefined and fixed during training, attention networks may learn redundant or even incorrect associations, thus affecting the understanding of objects and scenes. Therefore, our SSR introduces selective slot attention mechanism that dynamically adjusts the importance of slots for reconstruction. To quantitatively represent which slots are more important for reconstruction, we first introduce importance score for each slot. For each slot s,, we use a lightweight neural network he to predict an initial importance score: p! = o (heo(s))) where (.) is the sigmoid function, ensuring p. E [0. 1]. To capture interactions among slots, we further introduce an interaction matrix W E KRKK where each element Wiy is defined as Here, U and V are learnable projection matrices, and is We softmax, ((Us.) (Vs))/va) (VV,s)
scaling factor. The final importance score P: for each slot s; is then computed as P: = وド= Wispphit We then scale each slot representation by its respective importance score to create weighted slot representation $; Pt ss To this end, the overall training objective of SSR can be formulated as:
K
LSSR = || x
init
where is the balancing parameter. The regularization term encourages sparsity in the initial importance scores, guiding the model to utilize only the most relevant slots for reconstruction. On the other hand, the less relevant slots are down- wewn-weighted rather than discarded, which may help retain subtle information. Compared to continuous semantic representations, this training process is more manageable due to the discretization the slots. Since we lack truly fine-grained annotations, the learned slots "concepts" may not be readily interpretable as individual words. It is worth noting that incorporating strong visual feature extractor could significantly boost baseline performance. However, for the sake of fairness, we have opted not to use approaches, such as DINOSAUR [64].
3.2 High-Order Relational Reasoning (HORR)
Albeit we have obtained part-based representations in the form of slots, their inner- and inter- object relations are still under-explored. Prior graph-based based [9, 7] and non-grape onn-graph [33] works only accommodate pairwise relationships (e.g., connection between two nodes in a simple graph), but are inadequate to model the high-order relations inherent in images. Addressing this issue, we introduce a simple yet effective technique HORR- to model high-order topological relations with collection of graph nodes and hyperedges, divided into three stages: construction, learning, and matching.
Definition (Topological Homogeneity) We conceptualize topological homogeneity between the same object across images (hyper)graph matching problem, which is mathematically formulated as a relaxed quadratic assignment problem [52],
min || XBX 1
X |0,0 X1, < 1,, x 1m < Lns
where E Raxx and B Rm xm are the adjucent matrix encoding structure information of the graph GA and Gn respectively, n and m are the number of graph nodes, 1|r is the Frobenius norm, X,, E RRmxn is the unary affinity matrix and generally specified as the node affinity Mat, and X is the relaxed permutation matrix encoding mode-to-node assignment.

---

Hypergraph Construction. Given visual slot representations from the previous step, we carry out linear transformation obtain graph nodes VA and VB. This transformation is intended to map the visual features into the graph domain, ensuring effective graph matching. Then, we build hypergraph graphs 9사 and GB in both domains, modeling the topologiaal structures inherent in the images, IAA and p, respectively. The hypergraph [17] is defined as: GA/B (VA/B,&AJ &A/B W), where VA/B is the node sets, A is the hyperedge sets, and W E R| is diagonal matrix of edge weights. The hypergraph is denoted by an incidence matrix H E R Mix where H e) = 1 indicates the node E e and H(v, e) K-Nearest Neighbors (KNN), to establish hyperedge connections. The Euclidean distance is used to calculate the distance between node embeddings.
indicates & We adopt non-parametric density estimation, i.e.,
Hypergraph Learning- After constructing the hypergraphs, we update the graph node features according to the connectivity defined by the hyperedges. Technically, we introduce hypergraph convolution [17] to perform peessage-pasinng and feature aggregation,
= D 1/20
(5)
where @ represents the parameter to be learned during training. By doing so, can explicitly model intricate correlations among different parts of an image (i.e., slots). On the other hand, in contrast to grids, sequences, and even graphs, this high-order modeling can more effectively capture relations among nodes, avoiding excessive or erroneous pairwise connections.
Hypergraph Matching, Since the graph node features have been updated by Eq. (5), we introduce an affinity matrix aff to measure the node correspondence between GA and G3. Following [19, 44], we use the differentiable Sinkhorn layer [67] to calculate the affinity matrix W Waf. where each element indicates the degree of matching between pairs of nodes across graphs. To leverage the topological structures, we need a training objective to minimize the pairwise structural discrepancy between the follow the Definition 1 to specify unary affinity matrix X,, as
hypergraphs [78]. In particular, the obtained node affinity matrix Waff. Since the constructed hypergraphs naturally encode rich high-order relationships, they enhance the cross-domain topological matching process, Conversely, the matching process introduces additional structural knowledge to the current graph through message propagation. Formally, the hypergraph- -baee matching objective is formulated as follows:
Lmat =
Σ max Waf C Yn)L -1
+
Σ 1
Watt O (1 Ym) in
nnll
(6)
enhance true positive matches
penalize false positive matches
where the (i,j) element in YI E Ruxm is uz E GA and wff E GB belong to the same object class, otherwise it will O. Here, Yι can be regarded as pseudo labels for matching due to the absence of ground-trute truth correspondence. Moreover, as the matching primarily targets high-level semantic relationships, we avoid imposing additional structural constraints [23, 43, 44] on this training objective, thereby significantly simplifying the training process.
3.3 Training and Inference
Putting everything together, the full training objective formulated as follows:
CREMA = LcE + aLssR + 3.
(7)
where a and 3 are deep models using the reconstruction objective obtain discrete part representations with sufficient sparsity. Then turns to model the cross tross-image components correlations as the graph matching problem where further introduce the structural regularization term into the matching loss.
for balancing different loss terms. Our algorithm first trains
Note that our approach can be directly applied to test-time adaptation, similar to those self-super methods [69, 49, 3, 20]. The difference lies in the fact that their proxy tasks are somewhat heuristic, such as predicting rotations, jigsaw puzzles, and random masking. In contrast, our method directly exploits the intrinsic structured properties of the data, making more robust and versatile.
5

---

Table Comparison with OOD generalization methods on the PACS, Office-Home, and VLCS. Note that the results reported in this table do not involve any rest-time time adaptation strategies. Results are averaged over 3 random seeds. denotes the rounded standard error.
Algorithm
PACS
Office-Home
VLCS
Average Acc. (%)
ERM [72]
85.5
67.6
77.5
76.7
CORAL [68]
86.2
68.7
78.8
77.9
DANN [21]
83.7
65.9
78.6
76.1
MLDG [40]
84.9
66.8
77.2
76.3
CDANN [45]
82.6
65.7
77.5
75.3
MMD [42]
84.7
77.5
76.2
IRM [2]
83.5
64.3
78.6
75.5
GroupDRO [62]
84.4
66.0
76.7
75.7
I-Mixup [80, 82, 84]
84.6
68.1
77.4
76.7
RSC [28]
85.2
65.5
77.1
75.9
ARM [88]
85.1
64.8
77.6
75.8
MTL [5]
84.6
66,4
77.2
76.1
VREx [36]
84.9
66.4
78.3
76.5
Mixstyle [95]
85.2
60.4
77.9
74.5
SelfReg [32]
85.6
67.9
77.8
77.1
SagNet [55]
86.3
68.1
77.8
77.4
GVRT [53]
85.1
70.1
79.0
78.1
VNE [33]
86.9
65.9
78.1
77.0
REMA (Ours)
88.7
72.040.4
79.420.3
80.0
4
Experiments
In this section, we empirically evaluate the proposed REMA in two types of OOD scenarios, i.e., OOD generalization and test-time adaptation. In the following, we first describe the experimental setup (Section 4. and then provide the main results (Section 4.2) and ablation studies (Section 4.3).
4.1 Experimental Setup
Datasets. For OOD generalization, we leverage the three most widely used benchmark datasets. PACS [39] comprises 9,991 images and exhibits significant variations in image styles. It consists of domains each with classes, i.e., Photo, Art Painting, Cartoon, Sketch. VLCS con- tains 10,729 images of 5 classes from 4 photographic domains: PASCAL VOC 2007, LabelMe, Caltech, Sun. Office-Home [73] is collected from both office and home environments, and do- main shifts stem from variations in viewpoint and image style. It has 15,500 images of 65 classes from 4 domains, i.e., Artistic, Clipart, Product, Real World. Regarding test-time adaptation, we follow the common benchmarks [75, 30, 79] that utilize CIFAR-10/100 [35] and ImageNet [14] as the ID (training) data. CIFA R-10/100C [26] and Imagenet-C [26] are used as OOD (test) data, comprising different corruptions applied their original datasets.
Implementation Details. For OOD generalization, we use ResNet-50 for PACS, Office-Home, and VLCS and ResNet-18 for CIFAR-10. The model is trained using stochastic gradient descent with momentum 0.9, and weight decay 10-1, The training batch size set to 128. The learning rate 10-1. Following common practice, the model selection is based on training domain validation set. For test-time adaptation, we use ResNet-50 for all datasets. We utilize the Adam optimizer to update the network parameters. To facilitate a fair comparison, the test batch size of 64 in all methods. À in Eq. (3) is set to 0.01 all experiments. a and in Eq. (7) is set to 10 and 0.1, respectively.
We compare REMA against two types of baseline methods. (1) OOD Generalization: We adopt the onee-oon-dmmiinn evaluation protocol and follow the model selection strategy used in [25]. The main baselines have optimization-basee CORAL [68], MLDG [40], CDANN [45] and MMD [42], augmentation- Mixstyle [95], SagNet [55] and I-Mixup [80, 82, 84), and so (2) Test-time adaptation: All methods are based on online batch-level test data adaptation setting and the following baselines are included: entropy-minimization: Tent [75] and SHOT [47],
Baselines.
pseudo-labeling: T3A [29] and TAST [31], mansistency-aligmnenn TSD [79] and TIPI [56].

---

STM
-3-
78-
Cecotaon Node
misiio Hake
Herron
Zorn
heipinaal
Cencaas
Baei: Sanidoes
Pretate
Figure 3: Generalizing from CIFAR- 10 CIFAR-10C using ERM and our REMA.
Table 2: Comparisons with the state-of-the-e thr methods with average error rate (%) on image corruption benchmarks. Testing is conducted the highest level of image corruption. All methods use ResNet-50 backbone. J means lower is better.
Method
CIFAR-1OC CIFAR-100C
ImageNet-C
Avg. +
No Adaptation
29.1
60.4
82.0
57.2
+SHOT [47]
15.3
41.5
58.3
38.4
+Tent [75]
14.0
39.0
58.1
37.0
+PL [37]
22.3
40.1
63.0
41.8
+T3A [29]
26.7
58.3
75.8
53.6
+TAST [31]
26.6
60.7
+TAST-BN [31]
13.1
37.8
67.1
39.3
+TIPI [56]
13.5
38.3
55.9
35.9
+TSD [79]
13.1
37.7
53.2
34.6
+REMA
12.0
35.8
52.2
33.3
4.2 Main Results
OOD Generalization. The main results are presented in Table 1 and Figure 3. Notably, REMA consistently outperforms all baseline methods by large margin in each dataset. For example, compared to the recent method VNE, REMA improves classification accuracy by 1.8% for PACS, 6.1% for Office-Home, and 1.3% for VLCS. revealing the importance of our reconstruction (semantic abstraction) and matching (topological homogeneity) modules. Moreover, there are two notable observations: (1) Compared statistical matching methods (e.g., CORAL, DANN, and MMD) that directly optimize moment matching objectives in the latent space, REMA demonstrates superior performance by introducing topological information with more sparse input (slots). (2) Style or data augmentation based methods (e.g., MixStyle) is simple and to-immeementt.. However, their intuitive nature of interpolating around the training set may not accurately cover the target distribution region. By contrast, REMA directly learns major components from training data. (3) Compared to PACS and VLCS datasets, the Office-Home dataset has more categories and total samples. Many baseline approaches even perform worse than ERM, indicating their limited scalability. Instead, REMA demonstrates a larger performance gain in this challenging task.
Test-Time Adaptation. We compared REMA with existing advanced TTA methods, and the results summarized in Table 2 and Table REMA consistently provides improvements on multiple datasets. Compared to TSD [79]. it achieves 1. 1.3% improvement on three pixel-level corruption datasets and a 3.1% improvement on three challenging doos-nd datasets. Compared to the SOTA, REMA has three major advantages: (1) Tent [75] or SHOT [47] based on entropy minimization has limitations, as they can overcome distribution shifts caused by pixel corruption but fail handle cross-domain test data, even leading performance degradation. However, REMA can handle both types OOD data. (2) T3A [29] and TAST [31] based on pseudo-labels have performance limitations, and the pros and cons of pscudo-labels significantly impact the adaptation, which can be

---

Table Comparisons with the state-of-the- art methods on three image classification benchmarks.
Method
VLCS
PACS
Office-Home
Average Acc. (%)
ERM
76.7
83.2
67.1
75.3
+Tent [75]
73.0+1.3
85.2+0.6
66.3+0.8
74.9
+TentClf [75]
75.8
81.7
66.8+1.0
75.1
+SHOT [47]
67.1
84.1
67.610.7
72.9
+T3A [29]
77.3
83.9
68.3
76.5
+ TAST [31]
77.7
84.1
68.610.7
76.8
TAST-BN [31]
73.511.
89.2
68.910.5
77.2
+ TSD [79]
74.5
89.3
68 68.440.7
77.3
+REMA
79.4
90.3
71.640.6
80.4
4
Oseccton Modl that Bode
Inteu Bir
Bio
200f
FM
enpooue
Bastt Compresie
Test Data
w/o REMA
w/ REMA
oppate
(a) Grad-CAM Visualization
(b) Adaptation Order
Figure 4: (a) Visualization. (b) Analysis on continuous test-time adaptation.
observed in their ceilings on two types of datasets. REMA enhances resistance to noisy labels by capturing higher-order semantic dependencies. (3) REMA converts dense pixels into sparse slots, accurately capturing Nomain-invarian features, which will not be affected by pixel corruption and domain shift. Implementing consistent optimization during test time makes REMA more robust, resulting in 5.1% accuracy improvement across all benchmarks, iee. ERM REMA.
4.3 Ablation Studies
Table 4: Ablation of REMA (%).
Ablations of key modules in REMA. In this part, we provide the ablation results in Table 4,
ASR HORR
VLCS
PACS Office-Home
investigating the independent and combined ef-
fects of SSR and HORR proposed in REMA. As can be seen, incorporating SSR and HORR separately leads to improved generalization per- formance, demonstrating their contributions to abstraction and relational modeling. In addition, integrating SSR and HORR in method yields the best performance, highlighting that the two modules systematically work together and reciprocate each other. Moreover, as shown in Figure 4(a), the full REMA provide more complete and accurate representations of objects.
X
76.7
83.2
67.1
V
78.6
88.1
70.3
X
78.3
89.0
70.2
79.4
90.3
71.6
Analysis on continuous test-time adaptation. We empirically evaluate the continuous learning ability of REMA by comparing with state-of-the
Predicted Rosults
Ground Truth
test-time adaptation methods, namely Tent [75], EATA [57], and DeYO [38]. The results are pre- sented in Fig. 4(b), where the adaptation order from left to right. We can that the proposed REMA substantially and consistently outperforms all baseline methods as adaptation proceeds, revealing the robustness and the capability to handle open-world changes.
Analysis on hypergraph matching. In Figure 5, we visual- ize the learned doubly stochastic node affinity matrix and the ground-truth (GT) matrix. We observe that the proposed graph matching method effectively identifies the correct node affinity, revealing its effectiveness in handling cross-domain higher-order relationships
Figure 5: Learned affinity GT

---

(a) ERM
(b) REMA wio SSR
(c) REMA w HORR
(d) REMA (Tull)
Figure 6: Feature visualization of different methods using I-SNE,
Feature visualization. Figure 6 demonstrates the I-SNE [71] visualization of feature embeddings for ERM, REMA w/o SSR, REMA w/o HORR, and REMA (Full). We extract feature embeddings using CIFAR-1OC data, where the corruption type is snow with severity level of 5. The different color stands for different classes. We can observe that our REMA exhibits better clustering patterns with both SSR and HORR modules. When SSR removed, some closely related categories (e.g., animals) become highly mixed, indicating the advantage of SSR in extracting robust semantic representations. On the other hand, without HORR, intra-class variations increase, indicating some degree of ambiguity between different categories. Our HORR addresses semantic ambiguity by modeling topological homogeneity.
Related Work
OOD Generalization. The aim of OOD Generalization is to train a model using data from the source distribution so that it can perform well on unknown target distribution. A series of works can be divided into three categories: (1) Minimizing distribution differences [42, 45, 68, 90]: using meta-learning to simulate distribution shift [40, 18), or learning invariant transformation based on adversarial learning [42, 45, 89, 94, 93]. (2) Nomain-invariann representation learning: by analyzing the feature learning process of deep neural networks [16, 66, 27, 63, 12, 13], it is inferred how ID data can be generalized to OOD data [6, 65]. By understanding the interactions between ERM and generalization, domain-invariant invariant features are learned. (3) More generic OOD generalization: many studies focus on generalization under a relaxed assumption, such as open environments with unknown classes [8], semi-supervised settings in the testing environment [86], adapting foundation models [91], and OOD data synthesis [60, 59, 22], aiming to identify generalizable features from sophisticated test distributions. However, previous studies have only focused the instances themselves and have not explored the inherent high-order semantic dependencies in ID and OOD data, overlooking the topological homogeneity between distributions.
Test-time Adaptation. The goal of TTA [46] is to adjust the source trained model using test data without ground truth during the testing phase [47, 58, 10, 87, 79, 29]. Most of the existing methods directly perform gradient optimization on the test samples, with optimization objectives including prediction entropy [75]. self-training and stochastic restoring [77], or based normalization [54], etc., making the model adapt to the dynamic target environment at each step. Recently works further consider the scenario of noisy outlier samples [24, 70] appearing in the test stream. However, previous studies have simply optimized the test instances based on self-training, ignoring the topological relationship between samples and the high-order semantic dependencies within the test data steam, leading to suboptimal adaptation.
6 Conclusion
In this paper, we propose to achieve diit----distiittionn robustness from the perspective of ensuring topological homogeneity between the same object class regardless of the surrounding environments. To achieve this goal, we propose new framework REMA to first obtain a set of sparse and discrete representations from dense pixels and then model the high-order topological relations and depen- dencies via hypergraph (a generalized form of graph). Our experimental results reveal that REMA achieves superior performance on standard OOD generalization and test-time adaptation benchmarks. In the future, aim to extend this framework to more complex and dynamically changing scenes.

---

Acknowledgement
We thank the reviewers for their valuable comments. This work was supported in parts by NSFC (U21B2023, U2001206), ICFCRT(W2441020) Guangdong Basic and Applied Basic Research Foundation (2023B 155315151220 1512002)), DEGP Innovation Team (2022CXTD0025, Shenzhen Science and Technology Program (KQTD202 108 1 1090044003, RCJC20020071411445500 and Scientific Development Funds from Shenzhen University.
References
[1] Kartik Abuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In ICMI., pages 145-155, 2020.
[2] Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint wrXiv::907.02800 2019.
[3] Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. M13: Meta test-time training for self-supervised test-time adaption. International Conference on Artificial Intelligence and Statistics, pages 3080-3090. PMLR, 2022.
[4] Irving Biederman. Recooniition-y--oompooo review, 94(2):115, 1987.
a theory of human image understanding. Psychological
[5] Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research, 22(1):46-100, 2021.
[6] Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems, 35:25237-25250, 2022.
[7] Chaoqi Chen, Jiongeheng Li, Hong- Yu Zhou, Xiaoguang Han, Yue Huang. Xinghao Ding, and Yizhou Yu. Relation matters: Foreground-awer aware graph- based relational reasoning for domain adaptive object detection. TPAMI, 2022.
[8] Chaoqi Chen, Luyao Tang, Yue Huang, Xiaoguang Han, and Yizhou Yu. Coda: Generalizing to open and unseen domains with compaction and disambiguation. NeurlPS, 36, 2023.
[9] Chaoqi Chen, Luyao Tang. Feng Liu. Gangming Zhao Yue Huang, and Yizhou Yu. Mix and reason: Reasoning over semantic topology with data mixing for domain generalization. NeurIPS, 35:33302-33315, 2022.
[10] Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, pages 295-305, 2022.
[11] Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In CVPR, 2023.
[12] Liang Chen, Yong Zhang, Yibing Song, Anton Van Den Hengel, and Lingqiao Liu. Domain generalization via rationale invariance. ICCV, pages 1751-1760, 2023.
[13] Liang Chen, Yong Zhang, Yibing Song. Zhen Zhang, and Lingqiao Liu. A causal inspired carly-branching structure for domain generalization. IJC,, pages 1-21, 2024.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Puttern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 248-255. leee, 2009.
[15] Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for real-world domain generalization. CVPR, 2021.
[16] Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer. Tom Henighan. Shauna Kravec, Zac Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint 220: 10652, 2022.
[17] Yifan Feng. Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In AAAL volume 33, pages 3558 3565, 2019.
10

---

[18] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML, pages 1126-1135, 2017.
[19] Kexue Fu, Shaolei Liu, Xiaoyuan Luo, and Manning Wang. Robust point cloud registration framework based on deep graph matching. In CVPR, pages 8893-8902, 2021.
[20] Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. NeurIPS, 35:29374-22385, 2022.
[21] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Lavio- lette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 17(1):2096-200 2030, 2016.
[22] Jiahui Gao, Renjic Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. Self-guided noise-free data generation for efficient zero-shot learning. 2023.
[23] Quankai Gao, Fudong Wang, Nan Xue, Jin -ang Yu, and Gui-Song Xia. Deep graph matching under quadratic constraint. In CVPR, pages
2021.
[24] Taesik Gong, Yewon Kim, Taeckyung Lee, Som Chottamanurak, and Sung-Ju Lee. Sotta: Robust test-time adaptation on noisy data streams. Advances in Neural Information Processing Systems, 36, 2024.
[25] Ishaan Guirajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021.
[26] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[27] Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Syxtems, 33:9995-10006, 2020.
[28] Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV, pages 124-140, 2020.
[29] Yusuke Twasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. NeurIPS, 34:2427-2440, 2021.
[30] Minguk Jang and Sae-Young Chung. Test-time adaptation via self-training with nearest neighbor informa- tion. In ICLR, 2023.
[31] Minguk Jang, Sac- Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighbor information. In The Eleventh International Conference on Learning Representations, 2023.
[32] Dachee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jackoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In ICCV, pages
2021.
[33] Jacill Kim, Suhyun Kang, Duhun Hwang, Jungwook Shin, and Wonjong Rhee. Vne: An effective method for improving deep representation by manipulating cigenvalue distribution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3799-3810, 3810, 2023.
[34] Pang Wei Koh, Shiori Sagawa, Hearik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga. Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In ICML, pages 5637-5664, 2021.
[35] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[36] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation. In Interna- tional Conference on Machine Learning, pages 1-11615666666 2021.
[37] Dong- Hyun Lec. Pscudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML, volume 3, page 2, 2013.
[38] Jonghyun Lee, Dahuin Jung, Sachyung Lee, Junsung Park, Juhycon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In ICLR, 2024.
[39] Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV, pages 5542-5550, 2017.

---

[40] Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI, 2018.
[41] Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV, pages 1446-1455, 2019.
[42] Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR, pages 1400-5409, 2018.
[43] Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma: Semantic-complete graph matching for domain adaptive object detection. In CVPR, 2022.
[44] Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence. 45(7)9902-90000, 2023.
[45] Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV, pages 624 639, 2018.
[46] Jian Liang, Ran He, and Tieniu Tan. comprchensive survey on test-time adaptation under distribution shifts. arXiv preprint arv:v:2303. 15361, 2023.
[47] Jian Liang. Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML, pages 6028-6039 PMLR, 2020.
[48] Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie-Yan Liu. Learning causal semantic representation for out-of-distributtio prediction. NeurlPS, 34, 2021.
[49] Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Gurlet,, Taylor Mordan, and Alexandre Alahi. Ti++: When does supervid test-time training fail or thrive? NeurIPS, 34:21808 21820, 2021.
[50] Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf. and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. JCML, pages 41 44444, 2019.
[51] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. NeurIPS, 11111522-1338. 2020.
[52] Eliane Maria Loiola, Nair Maria Maia De Abreu, Paulo Oswaldo Boaventura- Peter Hahn, and Tania Querido. A survey for the quadratic assignment problem. European journal of operational research. 176(2))57-660, 2007.
[53] Sconwoo Min, Nokyung Park, Siwon Kim, Seunghyun Park, and Jinkyu Kim. Grounding visual represen- tations with texts for domain generalization. In European Conference on Computer Vision, pages 37-53, Springer, 2022.
[54] Zachary Nado, Shreyas Padhy, Sculley, Alexander D' A'mour,, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
[55] Hyconscob Nam, HyunJac Lee, Jongehan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR, pages 8690-8699 2021.
[56] A Tuan Nguyen, Thanh Nguyen-Tang, Tang, Ser-Nam Lim, and Philip HS Tomr. Tipi: Test time adaptation with transformation invariance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24162-24171, 2023.
[57] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICMI. pages 16888-16905. PMLR, 2022.
[58] Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh Ap. Generalization on unseen domains via inference-time time prese-ing target projections. In CVPR, pages 12933,, 2021.
[59] Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization. 2024.
[60] Renjie Pi. Jianshu Zhang, Jipeng Zhang, Rui Pan, Zhekai Chen, and Tong Zhang. Image textualization: An automatic framework for creating accurate and detailed image descriptions, 2024.
12

---

[61] Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via common- specific low-rank decomposition. In ICML, pages 7728-7738, 2020.
[62] Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural net- works for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations, 2020.
[63] Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller. Explainable Al: interpreting, explaining and visualizing deep learning, volume 11700. Springer Nature, 2019.
[64] Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Simon- Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf. Thomas Brox, et al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209. 14860, 2022.
[65] Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar. Data augmentation feature manipulation. In International conference on machine learning, pages 19773-19806 PMLR, 2022.
[66] Ravid Shwartz-Zi Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv: orXiv:/703 1703 108 2017.
[67] Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics, 35(2):876-879 1964.
[68] Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV, pages 443-450 2016.
[69] Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with suff-suupe for generalization under distribution shifts. In ICMI., pages 9229-9248. PMLR, 2020.
[70] Devavrat Tomar, Guillaume Vray, Jean-Philippe Thiran, and Behzad Bozorgtabar. Un-mixing test-time normalization statistics: Combatting label temporal correlation. arXiv preprint arXiv:2401 200.02080.0888 2024.
[71] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 9(11), 2008.
[72] Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks, 1089)9988-99, 1999.
[73] Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. CVPR, pages 5018 5027, 2017.
[74] Riccardo Volpi, Hongscok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurlPS, pages 5334-5344, 2018.
[75] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021.
[76] Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang. Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing unseen domains: A survey on domain generalization. TKDE, 2022.
[77] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7201-7211, 2022.
[78] Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and multiple-graph matching. IEFF TPAMI, 44()):5261-5279 2021.
[79] Shuai Wang, Daoan Zhang, Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In CVPR, 2023.
[80] Yufci Wang, Haoliang Li, and Alex C Kot. Heterogencous domain generalization via domain mixup. In IEEE International Conference on Acoustics, Speech and Signal Processing, pages
2020.
[81] Zehao Xiao, Xiantong Zhen, Shengcai Liao, and Cees GM Snock. Energy-based test sample adaptation for domain generalization. In ICLR, 2023.
[82] Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In AAAI Conference on Artificial Intelligence, pages 6502 6509, 2020.
13

---

[83] Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, and Mare Niethammer. Robust and generalizable visual representation learning via random convolutions. In ICLR, 2021.
[84] Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677 006777 2020.
[85] Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf. and Eric P Xing. Towards principled disentanglement for domain generalization. In CVPR, pages 8024-8034, 2022.
[86] Lei Zhang, Ji-Fu Li, and Wei Wang. Semi-supervised domain generalization with known and unknown classes. NeurIPS, 36, 2023.
[87] Marvin Mengxin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In NeurIPS, 2022.
[88] Marvin Mengxin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning adapt domain shift. In Advances in Neural Information Processing Systems, 2021.
[89]
Yahin Zhang. Bin Deng, Hui Tang, Lei Zhang, and Kui Jia. Unsupervised multi-class domain adaptation: Theory, algorithms, and practice. IEEE TPAMI, 2020.
[90] Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia, and Lei Zhang. Exact feature distribution matching for arbitrary style transfer and domain generalization. In CVPR, 2022.
[91] Yabin Zhang, Wenjic Zhu, Hui Tang, Zhiyuam Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: versatile adaptation approach for lannuaa models. In CVPR, 2024.
[92] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. TPAMI, 2022.
[93] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep bommain-advesariaa image generation for domain generalisation. AAAI, pages 13025-13032, 2020.
[94] Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. learning generate novel domains for domain generalization. ECCV, pages 561-578, 2020.
[95] Kaiyang Zhou, Yongxin Yang, Yu Qiao, Xiang. Domain generalization with mixstyle. In ICLR, 2021.
[96] Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In ICML. pages 27222-27244, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: The main claims the abstract and introduction accurately represent the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be pereeived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Iustification:
We provide this part in supplementary document.
Guidelines:
The answer NA means that paper has no limitation while answer means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust results are violations these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested a few datasets with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
grounds for rejection,
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
All assumptions and definitions have been clearly stated.
Guidelincs:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
15

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
Formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have fully disclosed all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
If the contribution is datasct and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, contribution novel architecture, describing the architecture fully might suffice, or the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require relcasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clcarly and fully.
(c) the contribution a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed-source source models, it may be that access to the model is limited in some way (c.g.. to registered users). but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code. with sufficient instructions faithfully reproduce the main experimental results, as described supplemental material?
Answer: [No]
Justification: After internal review and patent approval, we will release code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines (Ittpa://nips.cc
guides/Code CodeeS/odduubiii ssionPoiccy) for more details.
While we encourage release of code and data, we understand that this might not possible,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source henchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: :/ //nips...////////// cc/public/ guides wtw.le///oooeSuubmsssiiooeeeeeee
for more details.
16

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommeneed, but including URLs to data and code is permitted.
6. Experimental Setting/Detann Details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided all the training and test details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have provided the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We have provided these details in the supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
17

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, every respect, with NeurIPS Code Ethics https:/ httpp:///nurrpp.... cwtpublic/eeseeeeeeeecccc ceGuidelinig
Answer: [Yes]
Justification:
We have read the Code of Ethics and will always adhere to the principles it requires.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We have provided this part in the supplementary.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations. (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should deseribe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
18

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use publicly available datascts and code, and we have properly cited their papers or project addresses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, rememher to anonymize your assets (if applicable). You can either create an anonymized URL include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, other labor should be paid at least minimum wage in country data collector.
15. Institutional Review Board (TRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: [NA]
19

---

Guidelines:
answer means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
20


Reviews:
[{"review_id": "pxp5O6SQ9o", "reviewer": "Reviewer_jWEa", "summary": "A new methodology, REconstruct and MAtch (REMA) is introduced to learn a more robust and generalizable feature set in computer vision models.  REMA relies on a slot attention module to learn sparse embeddings of features which characterize a target object, and this module is then coupled with a high-order relational reasoning (HORR) module which creates a graph representation of the object.  This graph representation encodes how the sparse features relate to each other.  Because the feature encodings are both simplified through the sparse attention module and contain a learned topological relation from the HORR module, they are expected to be robust to domain drift. The robustness to domain drift is demonstrated using several benchmark datasets with consistent networks and training hyperparameters shared between them.  REMA consistently and clearly outperforms SOTA methods for OOD generalization; the results are convincing.", "questions": "1. The stated goal of the Selective Slot-based Reconstruction (SSR) module is to create a sparse embedding of target features.  In section 3.3, it would seem that a variational auto encoder type approach is used to train the encoder.  Sparsity can already be considered forced due to the size of the feature vector/latent space of the VAE; a more sparse feature vector is just one that is smaller in this context.  How does the additional SSR module change the latent representations to make them more sparse?  Is it just removing specific frequency components?  Why is it not possible to do this directly on the latent space of the encoder through a specialized loss function without the use of an additional module?  \n\n2. In section 3.1 \"(A) standard MLP skip connection is applied\" after the GRU; why? What is the intuition here?\n\n3. What is the minimum and maximum connectivity of the hypergraphs constructed in Section 3.2?  How would this dimensionality relate to the dimensionality of the sparse embeddings?  Of the original feature vector extracted from the encoding module?\n\n4. Please provide citation(s) for the statement in Supplemental section C.1 \"sparse modeling based on slots may struggle to accurately separate the scene into several main components\".  \n\n5. In the supplemental section B.2, why is it reasonable to expect that the SSR and HORR modules have different effects based on the type of data corruption?  Shouldn't the keypoints of a target object (and therefore the sparse embedding of those key points as well as the topology of how those keypoints relate to each other) be unaffected by whether it is fog or frost corrupting the image?", "limitations": "The authors report limitations in the supplemental material.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "The methodology is novel; the combination of the two modules represents a new contribution to the area of identifying robust features for computer vision.  The results reported are extensive, including latent space analysis of how features are represented, ablation studies, and comparison against $\\approx$20 other OOD generalization methods, with the proposed REMA method outperforming all of them.  An improved method for robust OOD model training is a significant contribution, and the work put into reporting the technical aspects of the methodology will encourage dedicated researchers to continue developing this approach to robust feature identification.", "weaknesses": "The report of the proposed method is exceedingly technical.  Discussion of the intuition motivating the method is restricted to a brief statement concerning the human visual system, and no connection back to the human visual system is made throughout the remaining discussion of the methodology.  No further motivation or context is provided for algorithmic design choices, which prevents the reader from understanding *why* the method works well. The emphasis is entirely on the *how*. I suspect that a significant literature review was performed as part of the algorithm design; sharing the literature review context for these design choices would have helped motivate and clarify the methodology.  It could easily have been included in the appendix without detracting attention from the main contribution of the work. Alternately, the context could have been included in the main result, and the implementation details left to the appendix.\n\nThe benefit of the HORR module is open to some question.  The impact of the HORR module vs the SSR module is discussed in Section 4.3, Table 4, and shown in Figure 6 with the tSNE embeddings.  The results of Table 4 are reported without uncertainty, and the values are close enough (within 2%-8.5% of the baseline without REMA) to make uncertainty a valuable indicator of the role of each module. Given the stochastic variation inherent to the tSNE algorithm, Figure 6c (SSR w/o HORR) and 6d (SSR w/ HORR) could be considered identical since there are no repeated embeddings reported, and no figure of merit describing the variance for repeated embeddings and clustering is included.  Visually, the embeddings are very close, so although the value of the SSR module is clearly demonstrated in disentangling latent features in Fig. 6c, the value of the HORR module has some doubt.  (An example of suitable metrics for quantifying the author's claim of better clustering in Figure 6d vs. Figure 6c might be DBSCAN or OPTICS applied to multiple embeddings, and quantified with the mean and std of the v-measure score for the clusters.)  The supplemental Figures 6-8 do little to clarify the situation; the authors state only that SSR and HORR behave differently depending on the type of corruption. An additional place to clarify the behavior of the individual SSR and HORR modules would have been in Figure 4a, where the grad cam results with and without the SSR and HORR modules could have been reported.  Finally, the comment in the supplement regarding training the SSR module first suggests that although including the HORR module may be beneficial, it is not necessarily key to the success of the work as no mention of training HORR first is made.\n\nOverall, the results are extensive, and support the conclusion that REMA is an improvement on the SOTA for OOD robustness.  But the lack of contextualization and motivation for the algorithm in general, and the behavior of the different modules specifically, is a weakness of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful comments and appreciation of our work. We address each question in detail and provide further clarifications below.\n\n> **Q1:Motivation for algorithmic design choices (SSR + HORR).**\n\n(1) Although we aim to imitate the human vision process for OOD generalization, we cannot guarantee that the learned components will perfectly align with concepts readily understood by humans, due to the lack of fine-grained human annotations. To solve this issue, our SSR is a data-driven approach that enables the deep model itself to possess the capability of abstraction from input data. The model essentially learns an attention mask, where each position's value indicates its importance/association relative to the class label. Given the need to learn objectness or high-level concepts, slot attention naturally comes to mind as a classic solution for object-centric learning. By doing so, as shown in **the attached PDF (Fig. 1)**, the regions segmented by SSR mostly align with human understanding, such as distinguishing between an animal's head and body. \n\n(2) Having identified the main components, we naturally consider the relationships among them. Graph networks are a major tool for introducing relational inductive bias. Given that slots are sparse and require higher-order connections to fully capture the relationships, and since ordinary graphs can only model pairwise relationships, we opt for hypergraphs. Moreover, regarding how to associate objects of the same category across different domains, most previous methods utilize direct alignment (Fig. 1 of the main paper). However, now that we have identified the main components and their internal relationships, and both have been modeled as vertices and edges of a hypergraph, cross-domain connections are naturally achieved via graph matching. \n\n> **Q2: The benefit of the HORR module.**\n\n(1) Standard error in Tab. 4. In fact, the results in Table 4 are averaged over 3 random seeds. Please see **the attached PDF (Tab. 1)**.\n\n(2) w/o HORR vs. w/ HORR. As suggested, we have run t-SNE 10 times for each case and then applied DBSCAN to each embedding. Then, we calculate the mean and std of V-measure score: w/o HORR (Mean V-measure: 0.65, std of V-measure: 0.08) vs. w/ HORR (Mean V-measure: 0.79, std of V-measure: 0.03). \n\n(3) Additional Gram-CAM. We added these results to **the attached PDF (Fig. 2)**. \n\n(4) Training Sequence. HORR first will lead to an unstable training process due to the dense latent features (without using SSR). Thus, joint training or SSR first would be better choices for our work.\n\n> **Q3-1: A VAE-type approach is used to train the encoder.**\n\nAs suggested, we trained an encoder using a VAE, using an equal number of mean and var as there are slots, for comparison. Please see **the attached PDF (Tab. 1)**. Noting that models with VAE achieve better performance than the ERM baseline, this validates our motivation to seek sparsity. However, there remains a performance gap compared to our SSR, demonstrating the superiority of our module.\n\n> **Q3-2: How does the additional SSR module change the latent representations to make them more sparse?**\n\nSSR actively reorganizes the latent space into discrete, interpretable slots, each capturing distinct and salient features of the input data. It structurally enhances sparsity by ensuring each slot is maximally informative and minimally redundant, thus facilitating sparse representations. This goes beyond just removing specific frequency components. Directly using a specialized loss function would lack this level of granularity and control. The modular nature of the SSR (data-driven) allows for targeted optimization and adaptation to diverse datasets.\n\n> **Q4: MLP skip connection.**\n\nThis is a standard step in slot attention, where (1) MLP allows the model to learn complex patterns and relationships between the slots and the input data. (2) Skip connections aid in preserving the original information from the input throughout the network.\n\n> **Q5: Details about hypergraphs and the dimensionality of embedding.**\n\nAs indicated in L143-144, the number of hyperedges is equal to the number of slots. The dimension of each slot is 256, matching the original feature dimension from the encoding module.\n\n> **Q6: Citation(s) for the statement in Supplemental section C.1**\n\nWe aim to divide objects into several components based on class labels. However, for scenes without clear foreground and background distinctions, this approach encounters challenges. In cases of tuberculosis—a binary classification problem—the X-ray images exhibit diffuse characteristics w/o specific lesions like lung nodules, which are causally linked to conditions such as malignant nodules and lung cancer. Thus, the difference is related to the image's style, making it hard to segment the original image into distinct parts based on the presence or absence of disease. While we did not find literature specifically addressing this issue, there are some related studies (e.g., [1,2]) that can be referenced.\n\n[1] Unsupervised Learning of Discriminative Attributes and Visual Representations. In CVPR, 2016.\n\n[2] Bridging the Gap to Real-World Object-Centric Learning. In ICLR, 2023.\n\n> **Q7: Why is it reasonable to expect that the SSR and HORR modules have different effects based on the type of data corruption?**\n\nSorry for the confusion. You are correct in pointing out that we aim to ensure that \"the keypoints of a target object—and therefore the sparse embedding of those keypoints as well as the topology of how those keypoints relate to each other—remain unaffected by external factors such as fog or frost,\" emphasizing topological homogeneity. We will revise this paragraph for clarity.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XTV7meSsbS", "reviewer": "Reviewer_M7aK", "summary": "The authors propose an approach to tackle OOD generalization via a method that tightly combines learning-based feature extraction with graph-based relationship modelling to explicitly learn and represent the topological structure of image data, achieving competitive results across a range of OOD and test-time adaptation benchmarks.", "questions": "**Main concerns, questions & potential improvements:**\n\n**[Q1]**: Consistency in notation could be improved.   \n The authors initially use lower-case characters to indicate functions, e.g. in equation (1), and mathcal font to indicate sets (e.g. input, output, feature space/domain);   \nHowever, this suddenly changes during the introduction of the method (l.94),  and mathcal is used to refer to transformations (e.g. K_beta, which is nothing else than a function);   \nThen slightly later (l.135), mathcal refers to the ‘result’ of the linear transformation that essentially forms the graph nodes;   \n$\\textrightarrow$ I’d highly recommend to keep a consistent notation to avoid confusion (as it did confuse me); \n\n**[Q2]**: Some missing details regarding training & inference procedure. \n- How exactly is training performed? The authors mention that the “algorithm first trains deep models using the reconstruction objective” (l.171) – is the model (or a part of it) then frozen when the graph is created/HORR trained, or is HORR simply added and everything then trained/finetuned? The appendix section doesn’t make this clear to me either, and some more details here would be helpful.\n- The authors mention in l.99 that the queries “will be refined during the T attention iterations”. How many iterations are employed in practice, and is this performed for each new image pair (i.e. each step) during training? \n- How exactly is the inference actually performed at test time? Does inference require pairs as employed during training, or what is the exact setup there? \n\n**[Q3]**: Details around complexity;   \nFollowing up from Q2, what is the ‘complexity’ of the method in contrast to other related methods, e.g. in terms of inference time?  \nNote that given the impressive performance and graph-based nature, this could be an interesting insight to the reader (to gauge potential trade-offs between multi- and 1-step reasoning methods)  \n\n**[Q4]**: Details regarding slots / information bottleneck:   \n- The authors mention in the appendix that ‘5’ slots have been typically used. Is this the same for all datasets? And how many were selected as relevant?\n- Do these numbers change across datasets or even classes? And if so, could you provide possible insights into why, e.g. multiple objects / increased complexity / more components, or similar.\n\n**[Q5]**: Insights into the actual 'slot correspondence'/representation:  \n- 5 slots seems quite few to represent the content of an image. Is this due to the simplicity of the images, and do the authors have some intuition how this would change for `real-world' natural settings? \n- How do the slots actually 'align' with 'components' of objects: Taking the motivational picture of the horse, would your algorithm actually identify 'parts' of an object as a component, or rather different objects in an image, or entirely different? Some insight into these aspects would be highly interesting to actually see to which extent the inner workings align with what we would expect from humans (which is, after all, your underlying motivation) \n\n---\n\n**Additional comments:**   \n- typo l.92: embedding -> embeddings \n- capitalization l.109: we -> We\n- typo l.216: methods […] is -> are", "limitations": "Limitations have been adequately addressed by the authors in the appendix;   \n$\\textrightarrow$ I highly appreciate the authors being honest and providing ‘proper’ limitations to their method in terms of potential applicability!", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality & Significance:**  \n- Interesting method focusing on explicitly modelling the often neglected /only implicitly modelled topological structure of image data via a mixture of learning- and graph-based methods\n- Proposed approach demonstrates an appealing way of ‘casting’ the common human intuition regarding our visual reasoning process, which draws on different levels of hierarchy, into an end-to-end trainable algorithm through a well-justified composition of different components\n\n**Quality:**  \n- Experiments conducted with a good selection of comparative methods to gauge performance improvements across two different tasks (OOD and test-time adaptation) and multiple datasets\n- The authors perform an appropriate ablation of their main components, both quantitatively and qualitatively \n\n**Clarity:**  \n- The paper is well written and easy to read and follow; The topic is well motivated and well-placed within the context of related efforts, clearly pointing out which areas of research the authors build upon and what the remaining challenge is;\n- Clear visualizations help the reader to quickly grasp the underlying concepts, both methodologically and architecturally", "weaknesses": "_TLDR; I do not see any severe ‘prohibitive’ weaknesses in this work, but have a few questions and requests that I’d like the authors to clarify & address._\n\n- Some insight into what the slots actually represent would be insightful and elevate the paper, see below.\n- Missing detail on complexity, please see question section below.\n- Missing details regarding inference and training procedure, see below.\n- Some inconsistencies in notation as well as some typos that should be corrected", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful comments and appreciation of our work. We address each question in detail and provide further clarifications below.\n\n> **Q1: Consistency in notation.**\n\nSorry for the confusion. We have modified the notation in Eq. (2) to make it consistent with other formulas, i.e., Q_gamma -> f_q, K_beta -> f_k, and V_phi -> f_v.\n\n> **Q2-1: How exactly is training performed?**\n\nWe employ a phased training strategy, starting with training SSR to enable the model to accurately extract sparse representations from images, followed by fine-tuning with HORR to equip the model with the capability to reason about these main components. The model (or a part of it) is not frozen during HORR training.\n\n> **Q2-2: How many iterations are employed in practice, and is this performed for each new image pair (i.e. each step) during training?**\n\nThe number of iterations is fixed at 3, consistent with the original slot attention mechanism. This process aims to progressively reconstruct the original image. In the context of slot attention using iterative attention, the process is generally applied repeatedly at each step during training for every new image pair. \n\n> **Q2-3: How exactly is the inference actually performed at test time? Does inference require pairs as employed during training, or what is the exact setup there?**\n\nDuring the inference phase, pairwise samples are not required; a single image or a batch of randomly sampled images suffices. For OOD generalization, a vanilla inference process is performed. For test-time adaptation, the model parameters need to be updated online using the test samples.\n\n> **Q3: Details around complexity.**\n\nWe have included quantitative results regarding the complexity of the method in **the attached PDF (Fig. 4 and Fig. 5)**. When the iterative attention and graph matching are only performed during training, the inference speed (without TTA) is nearly equivalent to ERM. With TTA **( Fig. 5)**, our method's inference speed is also comparable to several mainstream approaches.\n\n> **Q4: Details regarding slots / information bottleneck**.\n\nAs stated in the appendix, the initial number of slots is set to 5. In our experiments, this is the same for all datasets since OOD generalization and TTA benchmarks typically consist of single object images. Thus, there is no need to increase its value which will bring more computational overhead. However, for scene images, such as semantic segmentation datasets, we need more slots to present their composition. In a nutshell, the number of slots depends on the complexity of data and is relatively insensitive to the change of datasets that have the same type.\n\n> **Q5: Insights into the actual 'slot correspondence'/representation.**\n\nFor the first subquestion, **(1)** continuing from the previous discussion, the number of slots required is related not only to the complexity of the image content but also to the nature of the task itself. In our experiments, we focus on the relatively simple case of image classification tasks involving single objects, which require fewer slots. However, in scenarios containing multiple objects where tasks like object detection or semantic segmentation are desired, more than ten slots might be necessary, depending on the specific task. **(2)** It is important to note that more slots are not always better; on one hand, the computational cost increases, and on the other, there might be an over-segmentation issue in segmentation tasks (conversely, too few slots can lead to under-segmentation). However, for classification tasks, the granularity with which we recognize the main components of an object is flexible. For instance, in describing a person, we could (i) broadly identify the upper and lower body, (ii) recognize the head, torso, and limbs, or (iii) further divide specific parts such as the torso into more detailed segments. **(3)** In summary, classification problems require fewer slots and are less sensitive to the number of slots, whereas scene understanding tasks necessitate a larger and more sensitive allocation of slots. \n\nFor the second subquestion, **(1)** our process of identifying main components is somewhat akin to fully unsupervised attribute/concept discovery. That is, without component annotations, the model essentially learns an attention mask, where each position's value reflects its significance in relation to the class label. However, we cannot guarantee that the learned components will perfectly align with concepts readily understood by humans, as we lack fine-grained human annotations. Of course, replacing the encoder with a more powerful visual extractor like DINOv2 [1] could enhance discovery capabilities, but this would change both the whole experiment and the baseline methods. **(2)** From a methodological perspective, our proposed method can decompose images into high-level concepts in an unsupervised manner and cluster the images based on those discovered concepts. In **the attached PDF (Fig. 1)**, we provide some visual results from real-world datasets, showing that REMA can segment images into different areas (the number of areas depends on the number of slots). These areas are informative and correspond to different high-level concepts. For example, distinguishing between an animal's head, body, and legs. **(3)** As seen in Fig. 4(a) of the main paper, without REMA, the model might learn only small discriminative areas or even background regions. However, REMA enables the learning of objectness, which more completely emphasizes the entire foreground object area. In addition, the affinity matrix in Fig. 5 demonstrates that our REMA is capable of learning more accurate cross-domain correspondences, illustrating its robustness to distribution shifts.\n\n**Reference**\n\n[1] DINOv2: Learning Robust Visual Features without Supervision. In arXiv:2304.07193.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dgdgVlG6BP", "reviewer": "Reviewer_rhAB", "summary": "This paper presents REMA which designed to improve the robustness of deep learning models against out-of-distribution (OOD) data. REMA employs a selective slot-based reconstruction module to dynamically map dense pixels into a sparse set of slot vectors, enabling the identification of major components from objects in an unsupervised manner. Additionally, a hypergraph-based relational reasoning module is introduced to model high-order dependencies among these components, ensuring topological homogeneity. Experiments conducted on standard benchmarks demonstrate that REMA outperforms state-of-the-art methods in OOD generalization and test-time adaptation settings, highlighting its effectiveness in handling distribution shifts and enhancing the adaptability of deep models in real-world, non-stationary environments.", "questions": "1. Parameter Sensitivity: How sensitive is REMA's performance to the choice of hyperparameters such as the number of slots and attention iterations? Could an automated hyperparameter tuning method improve the robustness and generalization of the model further?\n\n2. Computational Efficiency: What is the computational overhead associated with the selective slot-based reconstruction and hypergraph-based relational reasoning modules? How does this overhead impact the scalability and real-time applicability of REMA in large-scale or resource-constrained environments?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. REMA introduces a unique combination of selective slot-based reconstruction and hypergraph-based relational reasoning to address OOD robustness, which has not been explored extensively in previous studies.\n\n2. The framework effectively identifies and leverages major components of objects without requiring human prior knowledge or fine-grained annotations, reducing the need for extensive labeled data.\n\n3. The paper provides extensive experimental results on multiple benchmark datasets, showing improvements over existing state-of-the-art methods in both OOD generalization and test-time adaptation scenarios.", "weaknesses": "1. The paper does not provide a detailed analysis of the computational overhead introduced by the new modules, which could impact the scalability of the approach for large-scale or real-time applications.\n\n2. While the experiments cover several benchmark datasets, the generalizability of REMA to other types of datasets or more diverse real-world scenarios is not fully explored.\n\n3. The effectiveness of REMA relies on several hyperparameters (e.g., the number of slots, attention iterations), and the paper does not thoroughly investigate the sensitivity of the model's performance to these parameters.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback, which we address below:\n\n> **Q1-1: How sensitive is REMA's performance to the choice of hyperparameters such as the number of slots and attention iterations?**\n\nAs suggested, we have provided quantitative experimental results in **the attached PDF (Fig. 3)**. Within a reasonable range, REMA is *not sensitive* to changes in hyperparameters (*max variation: ~1.8%*). Note that increasing the number of slots and iterative attention times will lead to higher computational costs. \n\n> **Q1-2: Could an automated hyperparameter tuning method improve the robustness and generalization of the model further?**\n\nIn our main paper, to ensure fairness in comparisons, we strictly follow the traditional DG methods for model selection. Introducing automated hyperparameter tuning could alter the experimental benchmarks. Moreover, searching a set of hyperparameters for each task would increase the training cost. As demonstrated in Q1-1, we empirically found that the proposed method is not sensitive to changes in hyperparameters. Therefore, we opted to fix hyperparameters across all experiments. On the other hand, it is true that this may not be the optimal solution for individual cases. Leveraging hyperparameter searching techniques to identify potentially optimal parameters for each case could further enhance the final performance. \n\n> **Q2-1: What is the computational overhead associated with the selective slot-based reconstruction and hypergraph-based relational reasoning modules?**\n\nIn **the attached PDF (Fig. 4 and Fig. 5)**, we have demonstrated the computational overhead of the selective slot-based reconstruction (SSR) and hypergraph-based relational reasoning (HORR) modules. (1) During training, compared to previous OOD generalization methods, our SSR, which utilizes a small number of slots, does not significantly increase computational costs; HORR also does not incur substantial overhead due to the limited number of hypergraph nodes and edges. Thus, our method (SSR + HORR) is close to previous methods in terms of param. and GFLOPs **(Fig. 4)**. (2) During inference, the speed of our method is comparable to most previous TTA methods **(Fig. 5)**. Note that without TTA, it is equivalent to the inference speed of ERM.\n\n> **Q2-2: How does this overhead impact the scalability and real-time applicability of REMA in large-scale or resource-constrained environments?**\n\n(1) Scalability. First, the time complexity of Slot Attention is approximately O(N×K×d), which indicates that the computational cost of the algorithm increases linearly with the increase in the length of the input sequence, the number of slots, and the feature dimensionality. In our case, N and K are generally O(1), and d is typically O(10^2), thus the complexity of hypergraph convolution is approximately O(10^2). Second, for hypergraph convolution, the time complexity is often in the order of O(E×C×d), where E is the number of hyperedges, C is the average cardinality of the hyperedges (i.e., the average number of vertices that each hyperedge connects), and d is the dimensionality of the features. Similarly, E and C are generally O(1), and d is typically O(10^2), thus the complexity of hypergraph convolution is approximately O(10^2). Based on this detailed analysis of complexities, we find that the two main operations in REMA are efficient, indicating the potential to scale up. \n\n(2) Real-time applicability. In our main paper (Sec. 4.3 and Fig. 4(b)), we show the results of continuous test-time adaptation, which aims to evaluate the capability to handle dynamically changing environments. Our method consistently outperformed SOTA methods, showing superior stability without significant changes. As mentioned in Q2-1, our computational overhead is not substantial. Therefore, combined with our capability for continuous adaptation, this ultimately allows us to meet real-time demands—ensuring both speed and stability.\n\n> **Q3: The generalizability of REMA to other types of datasets or more diverse real-world scenarios.**\n\nThank you for your suggestion. As suggested, we have added two typical applications in medical scenarios, including pneumonia classification (chest X-ray images) [1] and skin lesion classification [2]. Please refer to the original papers for details of the experimental setup. The results are presented below.\n\n**Pneumonia Classification.** Chest X-ray images from three different sources: NIH, ChexPert, and RSNA. The task is to detect whether the image corresponds to a patient with Pneumonia or not. \n\n| Method      | RSNA | ChexPert | NIH  |\n| ----------- | ---- | -------- | ---- |\n| ERM         | 55.1 | 60.9     | 53.4 |\n| IRM         | 57.0 | 63.3     | 54.6 |\n| CSD         | 58.6 | 64.4     | 54.7 |\n| MatchDG [1]   | 58.2 | 59.0     | 53.2 |\n| MiRe [3]    | 63.6 | 65.0     | 56.4 |\n| REMA (Ours) | 68.2 | 70.5     | 62.4 |\n\n**Skin Lesion Classification.** We adopt seven public skin lesion datasets, including HAM10000, Dermofit (DMF), Derm7pt (D7P), MSK, PH2, SONIC (SON), and UDA, which contain skin lesion images collected from different equipment. \n\n| Method      | DMF  | D7P  | MSK  | PH2  | SON  | UDA  | Avg  |\n| ----------- | ---- | ---- | ---- | ---- | ---- | ---- | ---- |\n| DeepAll     | 24.9 | 56.8 | 66.7 | 80.0 | 86.1 | 62.6 | 62.9 |\n| MASF        | 26.9 | 56.8 | 68.2 | 78.3 | 92.0 | 65.4 | 64.6 |\n| MLDG        | 26.7 | 56.6 | 68.9 | 80.2 | 88.2 | 63.2 | 64.0 |\n| CCSA        | 27.6 | 57.4 | 68.3 | 75.0 | 90.5 | 67.6 | 64.4 |\n| LDDG [2]      | 27.9 | 60.1 | 69.7 | 81.7 | 92.7 | 69.8 | 67.0 |\n| REMA (Ours) | 29.1 | 63.4 | 70.8 | 83.5 | 94.8 | 75.3 | 69.5 |\n\n**References**\n\n[1] Domain Generalization using Causal Matching. In ICML, 2021.\n\n[2] Domain Generalization for Medical Imaging Classification with Linear-Dependency Regularization. In NeurIPS, 2020.\n\n[3] Mix and Reason: Reasoning over Semantic Topology with Data Mixing for Domain Generalization. In NeurIPS, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vCxQAPw1n0", "reviewer": "Reviewer_EaaQ", "summary": "The paper proposes a method for improving domain generalization and test time adaptation by introducing a selective variant of slot attention, formulating the relationship between slots across images as topological homogeneity between hypergraphs constructed based on the slots, and thereby matching occurances of the same object between different images.", "questions": "The paper requires a major revision to make it much more clear which setting the authors are targetting: Is it:\n- OOD generalization as suggested by the title? Then, the authors **cannot** assume any knowledge about the distribution shift at test time. Then the baselines in Table 1 cannot be used as they do assume access to the target domains.\n- Unsupervised Domain Adaptation as suggested by the baselines in Table 1, i.e. DANN and CORAL? Then, the authors need to reformulate their title, abstract, motivation etc. and remove all instances of \"OOD generalization\" because this is very confusing otherwise. The authors also need to remove the Domain Generalization baselines from Table 1.\n- Domain Generalization as suggested by the IRM and VNE baselines in Table 1? Then, the authors need to remove the UDA baselines from their Table 1 and remove instances of mentioning OOD generalization from the paper.\n\n\nThe authors need to add Test-Time adaptation baselines to their Table and improve their continual Test-Time evaluation.\n\nThe authors need to make more comprehensive evaluations on the influence of the different components of their loss.", "limitations": "The limitations section in the appendix is very short. A clear limitation is that the method introduces several hyperparameters which need tuning. This imposes an additional computational overhead which has not been discussed.\n\nAnother limitation which can be fixed is that the authors did not study the continual learning setting to analyze whether their method collapses when adapting for long time periods.\n\nIn 2024, most people work with large-scale pretrained models such as e.g. CLIP. It is not clear how this method could be used on those models and whether improving ResNet50's performance is relevant nowadays.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method and its components introduce novel techniques for the domain generalization problem that are well motivated and appear to be sound.\n\nThe mathematical derivation of the employed forms of the individual steps makes sense.\n\nSeveral ablations well demonstrate the dependence of REMA's results on its main building blocks.", "weaknesses": "The proposed method consists of fairly many newly introduced steps.\nWhile ablations show their necessity of the major modular building blocks of REMA for achieving the eventual results, these building blocks are not analyzed in detail.\nFor understanding them (and therefore REMA), it would be necessary to expound each individual step and confirm that it has the intended effect and that the explicit or implicit mathematical assumptions hold.\nFor example, the results of standard slot attention can be of varying quality, and it would be important to verify whether and how such imprecisions introduced there propagate through the next steps.\nThe analysis in Figure 5 goes in this direction, but it is not clear what exactly is probed and measured here. A similar analysis as in for the other components of the method would be helpful.\n\n**Main issue of the paper**: There exist different communities in ML which study the generalization performance of DNNs to distribution shifts in very different settings. Unsupervised Domain Adaptation (UDA) is separate from Domain Generalization (DG) and both are separate from OOD generalization. Putting papers which study either of those in the same Table is wrong, misleading and highly confusing. All of these settings have their benefits and challenges and the numbers are simply incomparable if the initial conditions, such as which data is available during training, differ this strongly. The authors conflate and confuse all three settings on multiple occasions which makes the paper confusing and the results are impossible to understand. It is further completely unclear in which of the three their proposed method falls into. Below, I explain the issue in much more detail and provide specific problematic text instances. The paper needs a major revision which includes reworking all sections and most Tables.\n\nline 183: \"For OOD generalization, we leverage the three most widely used benchmark datasets.\" The datasets PACS and Office-Home are usually used in Unsupervised Domain Adaptation (UDA), not OOD generalization. Datasets commonly used to benchmark OOD generalization are datasets like ImageNet-R, ImageNet-Sketch, ObjectNet or ImageNet-V2. The choice of the baselines the authors compare to also gives the impression that they confuse domain adaptation with OOD generalization: CORAL and DANN are UDA methods.\n\n\nline 66: \"The goal of OOD generalization is to find a predictor f : X 7→ Y that generalizes well to all unseen target domains.\" This is correct, but then the used baselines are wrong because they assume access to the target domains and are trained using this information. It is not clear to me how the UDA methods (shown in Table 1) can be implemented without this access.\n\n\nThe setting of UDA differs drastically from OOD generalization: In UDA, we attempt to learn a model which performs well on the source domain and a set of target domains, using unlabeled data from the target domains. That is, in UDA, we assume access to the unlabeled test set at training time. This differs from the OOD generalization setting: Here, we want to train a model on a source dataset and then test its generalization to an unseen test set. That is, we assume no knowledge about the test time distribution shift. This is a crucial difference and it is wrong to confuse the two terms. The implemented baselines, e.g. CORAL and DANN, are used in a UDA setup, not in an OOD generalization setup. In this light, the wording \"Following common practice, the model selection is based on a training domain validation set\" is meaningless because the common practice for model selection differs depending on which setting we consider.\n\n\nThe related work section is similarly confusing. On the one hand, the authors wish to review works on OOD generalization where the goal is \"to train a model using data from the source distribution so that it can perform well on an unknown target distribution\". [41] and [42] use the DG setting, i.e. assumes access to n-1 target domains. [63] assumes access to all target domains. It is unclear why [61] is a good citation for domain-invariant learning. I haven't checked all citations, but the ones I checked **all** assumed access to the target domains.\n\n\nThe setting the baselines use in Table 1 is inconsistent. CORAL and DANN have been trained in an UDA setting. That is, they assume access to unlabeled target data at test time from a target domain. VNE and IRM are used in a domain generalization setting where they assume access to n-1 domains and aim to perform well on an unseen domain. These are two very different settings which are incomparable to each other and it is wrong and confusing to put them in the same table without any discussion. Further, the authors aim to test OOD generalization where no access to unseen target domain data is available (line 66) which is incompatible with either UDA or domain generalization.\n\n\nThe baseline numbers for the cited papers are wrong. I checked the numbers of CORAL and VNE from Table 1 with the numbers in the original papers and they do not match. If the authors reimplemented all the baselines, they need to state this and discuss where the performance differences come from.\n\n\nChecking papers-with-code for the PACS dataset (https://paperswithcode.com/sota/domain-generalization-on-pacs-2), the best numbers are close to 100% for the domain generalization setting. Since the authors use DG benchmarks in their table 1, I assume that using other benchmarks in the DG domain would be valid as well. If we filter for RN50 architectures only, the best number is 90.5% which is higher compared to the best number reported by the authors.\n\n\nLooking at the Office-Home dataset (https://paperswithcode.com/sota/unsupervised-domain-adaptation-on-office-home), the best number here is 90%. Here, the setting is UDA. But as mentioned before, it is not clear which setting should be used because the authors include both UDA and DG benchmarks in their Table 1. The best number in the UDA setting is higher than what the authors report in Table 1. There are actually five separate benchmarks for OfficeHome on papers with code (Domain Generalization, Domain Adaptation, Universal Domain Adaptation, UDA, Partial DA) and it is not clear which one should be used.\n\n\nIt is not clear how the models are trained. The test sets are comprised of different domains. Do the authors train their models on all of them or on e.g. two of the domains and test against the rest or on one of the domains and then test against the rest?\n\n\nFor test-time adaptation, the authors missed important works which perform better than their method. SLR achieves an error of 48.7% on the highest severity of ImageNet-C [A]. ETA and EATA [B] achieve an error of around 52%. \n[A] Mummadi et al. \"Test-time adaptation to distribution shift by confidence maximization and input transformation\"\n[B] Niu et al. \"Efficient test-time model adaptation without forgetting.\"\n\n\nPress et al. [C] showed that most test-time adaptation methods collapse when adapting for long periods of time. The authors should test their model on the proposed CCC benchmark to analyze whether their method also suffers from the collapse. The authors are writing that they test their method in  the continuous test-time adaptation scenario in Fig. 4b, but it is entirely unclear on which dataset they are testing it. Is it the CCC benchmark? Notably, if the authors are testing their approach on the Continual Test-time adaptation benchmark from [D] (although I am just guessing here), Press et al. showed in [C] that one needs to adapt for longer time periods to show-case collapse and the adaptation periods proposed in [D] are insufficient. The authors must state clearly which continual learning benchmark they are using and I would suggest to report numbers on both [C] and [D].\n[C] Press et al. \"RDumb: A simple approach that questions our progress in continual test-time adaptation\"\n[D] Wang et al. \"Continual test-time domain adaptation\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the time and effort the reviewer has invested in reviewing our paper. However, **we must point out that the review contains many factual errors and misunderstandings, rendering most comments unacceptable due to their erroneous assumptions.** We will try our best to eliminate the misunderstandings via the following responses.\n\n> **Q1: Regarding problem definitions and experimental comparisons**\n\n(1) Our paper is NOT related to the UDA setting; we consistently emphasize our research on OOD generalization (a.k.a. domain generalization). The reviewer's comment that \"Unsupervised Domain Adaptation (UDA) is separate from Domain Generalization (DG) and both are separate from OOD generalization\" is **problematic**. In our paper, **DG and OOD generalization are synonymous and can be used interchangeably**. This is also a **consensus** within the community, as exemplified by two well-known DG surveys [1,2] that treat the terms as synonymous. In the abstract of [1], it is stated: \"Domain generalization (DG), i.e., out-of-distribution generalization, has attracted increasing interest in recent years.\"\n\n(2) In Table 1, we ONLY compare with DG baselines. The reviewer's claim that our baseline comparisons, which include DANN and CORAL, suggest we are evaluating UDA algorithms is a **significant misunderstanding**. DANN and CORAL, originally introduced as UDA methods, are now important DG baselines that match deep features from different source domains (NOT target domain) via domain adversarial training (DANN) and correlation alignment (CORAL). This is a **common practice in the DG community**. The well-known **DomainBed** benchmark (code: https://github.com/facebookresearch/DomainBed) and many follow-up works in the field also include both DANN and CORAL as their main DG baselines. Regarding the baseline results, we cite all these numbers from DomainBed or their original papers (following the identical setting) to ensure a fair comparison. Our experiments were also conducted in the same conditions. Please refer to recent papers to avoid misunderstandings regarding benchmarks, such as [3-6]. Based on this, we need to emphasize that the comparisons in Table 1 are reasonable and fair, and our problem definition is also precise and clear.\n\n(3) The reviewer's statement that ''The baseline numbers for the cited papers are wrong. I checked the numbers of CORAL and VNE from Table 1 with the numbers in the original papers and they do not match'' is **incorrect**. Regarding CORAL, its original paper conducted UDA (not DG) experiments on Office-31. **Moreover, it did not involve any experiments on PACS, Office-Home, or VLCS at all. Why then is it claimed that our ''cited'' results are wrong?** Regarding VNE, we used the results from Table 2 of the original paper, which were obtained using the **ERM** algorithm (see our Table 1). The results from Table 3 of the original VNE paper, which were obtained using the **SWAD** algorithm, are inconsistent with our experiments and thus cannot be used. Regarding the mentioned results from paperswithcode website, it is not standard practice in the DG community to base comparisons on them due to the potential distinction in training and inference. For example, the paper that achieved 90.5% on PACS employed a different model selection strategy (not ID val).\n\n(4) The reviewer's statement that ''the PACS and Office-Home datasets are usually used in Unsupervised Domain Adaptation (UDA) and not OOD generalization'' is **incorrect**. **The PACS dataset was introduced in the paper \"Deeper, Broader and Artier Domain Generalization,\" which is a DG paper, not UDA.** Actually, the datasets we use are among the most commonly utilized in the DG community for image data, as referenced in [1-6].\n\n(5) We clarify why our paper also conducted experiments related to test-time adaptation (TTA). The goal of TTA [7] is to update the model online during testing, which complements the goal of DG, i.e., DG aims to learn a generalizable model using only source data, while TTA seeks to enhance generalization ability using unlabeled target data. On the other hand, a series of TTA works utilizes self-supervised learning tasks for both the training and testing phases (L174-177). This aligns with our slot-based approach, and our hypergraph-based matching naturally links training and test phases, enabling our framework to also work as a TTA method. We reiterate that the experiments for DG and TTA were conducted separately (Tab. 1 does not utilize any TTA strategies).\n\n**References**\n\n[1] Generalizing to Unseen Domains: A Survey on Domain Generalization. TKDE, 2022.\n\n[2] Domain Generalization: A Survey. TPAMI, 2022.\n\n[3] Diverse Weight Averaging for Out-of-Distribution Generalization. NeurIPS, 2022.\n\n[4] Improving Out-of-Distribution Generalization by Adversarial Training with Structured Priors. NeurIPS, 2022.\n\n[5] MADG: Margin-based Adversarial Learning for Domain Generalization. NeurIPS, 2023.\n\n[6] On the Adversarial Robustness of Out-of-distribution Generalization Models. NeurIPS, 2023.\n\n[7] A Comprehensive Survey on Test-Time Adaptation under Distribution Shifts. IJCV, 2024.\n\n> **Q2: Regarding TTA experiments**\n\n(1) We thank the reviewer for pointing out these two works [A, B] and have added them to our paper. However, we have to note that SLR [A] is still a preprint paper (since 2021), uses extra input augmentation (compared to other baselines), and does not release code (hard to make a fair comparison).\n\n(2) Regarding continuous TTA, we have already presented it in Fig. 4(b) (CIFAR-10C), where the x-axis represents the type of corruption and multiple SOTA TTA methods are compared. More results (CIFAR-100C and ImagNet-C) will be added to the final version.\n\n> **Q3: Regarding ablation studies**\n\nIn the original manuscript, the ablation of REMA is shown in Tab. 4, Fig. 4, and Fig. 6. As suggested, we provide additional ablation studies in **the attached PDF (Tab. 2)**.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate all four reviewers for their time and effort in providing feedback and suggestions on our work. We are glad that reviewers recognize our paper to be *novel* (jWEa, M7aK), *well-motivated* (M7aK, rhAB, EaaQ), and performing *extensive experiments and ablation studies* (jWEa, M7aK, rhAB). 

We have addressed the comments and questions in individual responses to each reviewer. The main changes we made include:

- We have provided additional visual results and experimental comparisons, explained the design motivations of our algorithm, and offered insights into the proposed SSR and HORR (jWEa).
- We clarified some design details, discussed complexity, and provided insights into the actual 'slot correspondence' and representation (M7aK).
- We conducted both quantitative and qualitative discussions on parameter sensitivity, computational efficiency, and performance in real-world scenarios (rhAB).
- We provided clarifications to eliminate significant misunderstandings about the problem definition, baseline selection, and experimental comparisons (EaaQ).

If you have any further questions or require additional clarification, please feel free to raise them during the author-reviewer discussion phase. Thank you!

Meta-review: This paper received an Accept, two Weak Accepts, and a Borderline Accept. All reviewers agree that the work presents an interesting and innovative approach. Notably, they commend the novelty and well-motivated design of the selective slot-based reconstruction module and the hypergraph-based relational reasoning framework. After the rebuttal, the authors successfully addressed all the concerns and questions raised by the reviewers, resulting in all reviewers being satisfied post-rebuttal. Consequently, the AC has decided to Accept this paper, recognizing its potential impact on both the theoretical and practical aspects of out-of-distribution generalization tasks.
Decision: Accept (spotlight)