Title: Reconstruct and Match: Out-of-Distribution Robustness via Topological Homogeneity
Venue: NeurIPS 2024
Accepted: yes
Keywords: Out-of-Distribution

Domain Generalization

Test-Time Adaptation

Topological Homogeneity
Abstract: Since deep learning models are usually deployed in non-stationary environments, it is imperative to improve their robustness to out-of-distribution (OOD) data. A common approach to mitigate distribution shift is to regularize internal representations or predictors learned from in-distribution (ID) data to be domain invariant. Past studies have primarily learned pairwise invariances, ignoring the intrinsic structure and high-order dependencies of the data. Unlike machines, human recognizes objects by first dividing them into major components and then identifying the topological relation of these components. Motivated by this, we propose Reconstruct and Match (REMA), a general learning framework for object recognition tasks to endow deep models with the capability of capturing the topological homogeneity of objects without human prior knowledge or fine-grained annotations. To identify major components from objects, REMA introduces a selective slot-based reconstruction module to dynamically map dense pixels into a sparse and discrete set of slot vectors in an unsupervised manner. Then, to model high-order dependencies among these components, we propose a hypergraph-based relational reasoning module that models the intricate relations of nodes (slots) with structural constraints. Experiments on standard benchmarks show that REMA outperforms state-of-the-art methods in OOD generalization and test-time adaptation settings.

Reconstruct and Match: Out-of-Distribution Robustness via Topological Homogeneity

Abstract

Since deep learning models are usually deployed in non-stationary environments, it is imperative to improve their robustness to out-of-distribution (OOD) data. A common approach to mitigate distribution shift is to regularize internal representations or predictors learned from in-distribution (ID) data to be domain invariant. Past studies have primarily learned pairwise invariances, ignoring the intrinsic structure and high-order dependencies of the data. Unlike machines, humans recognize objects by first dividing them into major components and then identifying the topological relation of these components. Motivated by this, we propose Reconstruct and Match (REMA), a general learning framework for object recognition tasks to endow deep models with the capability of capturing the topological homogeneity of objects without human prior knowledge or fine-grained annotations. To identify major components from objects, REMA introduces a selective slotbased reconstruction module to dynamically map dense pixels into a sparse and discrete set of slot vectors in an unsupervised manner. Then, to model high-order dependencies among these components, we propose a hypergraph-based relational reasoning module that models the intricate relations of nodes (slots) with structural constraints. Experiments on standard benchmarks show that REMA outperforms state-of-the-art methods in OOD generalization and test-time adaptation settings.

1 Introduction

Distribution shift has emerged as a major challenge for the success of machine learning systems [34, 25]. Although deep learning models are believed to generalize well to in-distribution (ID) data, a well-trained model deployed in the open world often encounters out-of-distribution (OOD) data whose contexts may differ from the training distribution, resulting in dramatic performance degeneration and raising concerns about model safety in many high-staking applications, such as autonomous driving and medical diagnosis. This gives rise to the importance of OOD generalization [92, 76], which aims to build a robust learning machine that can perform well in unseen test environments.

Regarding OOD generalization, a central theme is how to learn general features from training data that can be extrapolated to test distributions. Following this idea, a plethora of OOD generalization methods have been proposed over the past few years, including domain alignment [42], latent feature disentanglement [61, 48, 85], meta-learning [40, 41], invariant risk minimization [2, 1, 96], and augmentation-guided invariant predictor [74, 83, 95], to name a few. On the other hand, given the natural adaptivity gap between training and test distributions [15], recent works [29, 30, 81, 11, 8] attempt to further enhance the source-trained model through test-time adaptation [46], which leverages unlabeled target samples to update the model in an online manner. These two series of studies address distribution shifts during training and inference and have been proven to work synergistically [8].

Disentanglement

Dense & Continuous

Sparse & Discrete

Reconstruction

(a)

Idea case

(b)

Popular approach

(c)

Our framework

Albeit their general efficacy for various tasks, these prior efforts largely overlook the topological structure of the image data. As a consequence, existing OOD generalization methods highly rely on specialized regularization objectives and are significantly less interpretable than human vision systems. Unlike machines, human object recognition [4] typically involves initially decomposing objects into several major components ( e.g., keypoints), followed by identifying the structural relationships among these components, and finally making predictions by comprehensively considering the main components and their inherent relations. To this end, a critical question remains open in the field:

How to devise a unified framework that imitates the human vision process for OOD generalization?

In this work, we believe that OOD generalization requires consideration of two key aspects: 1) how to represent the structure of data, and 2) how to model the relationships between different entities.

Grounded on these insights, we propose Reconstruct and Match (REMA), a general framework to endow deep models with the capability of capturing the topological homogeneity of objects without using human prior knowledge or fine-grained attribute annotations. Figure 1 illustrates the motivation. To identify major components from objects, REMA introduces a slot-based reconstruction module to map dense pixels into a sparse set of slot vectors in an unsupervised manner. This module encourages the deep model to reduce unwanted redundancy and preserve those predominant parts (the words 'component' and 'part' are used interchangeably). Then, to model and reason the highorder dependencies among these components, we propose a hypergraph-based matching module that discovers the relations of slots with structural constraints, i.e., topological homogeneity of the object. The main contributions of this paper are summarized as follows:

We propose REMA, a novel OOD generalization framework to mitigate distribution shifts in deep learning models by imitating the human visual recognition process.

We introduce a self-supervised reconstruction process to identify informative parts from objects without additional supervision, and a hypergraph matching module to reason the high-order part-based object relationships and interactions across domains.

Experiments on six widely used benchmarks demonstrate that REMA outperforms state-ofthe-art methods in OOD generalization and test-time adaptation settings.

2 Preliminaries

Problem Setup. Assume that X is the input space, Z is the latent space, and Y is the output space. The predictor f = h ◦ g is comprised of a featurizer g : X ↦→ Z that learns to extract embedding features, and a classifier h : Z ↦→ Y that makes predictions based on the extracted features. The goal

Encoder

Input

MLP

Extracted

Features

Reconstructed

Features

Slot Selection

HGNN

Visual-to-Graph

Hyperedge

Sinkhorn

Layer

Slot

Attention

High-Order Relational Reasoning (HORR)

Linear

Transform

HGNN

Optimum

ℒ𝑚𝑎𝑡

ℒ𝑆𝑆𝑅

of OOD generalization is to find a predictor f : X ↦→ Y that generalizes well to all unseen target domains. In deep neural networks, empirical risk minimization (ERM) [72] is capable of learning highly predictive features, making it the simplest baseline for this problem.

ERMObjective. We train the model by minimizing the empirical cross-entropy loss function:

̸

where P tr is the training distribution, Θ denotes the parameter space, and ℓ is the loss function. The trained model will be evaluated on a test set from a test distribution P ts , where P tr = P ts .

3 Proposed Method

In this section, we provide a detailed description of our Reconstruct and Match (REMA) framework. As shown in Figure 2, REMA consists of two key modules tackling the main component discovery (Section 3.1) and high-order relational modeling and reasoning (Section 3.2).

3.1 On Discovering Main Components

For OOD generalization/adaptation, a long-standing issue is how to distinguish between transferable (domain-agnostic) and non-transferable (domain-related) information in the data. An ideal solution is to disentangle latent representations in an unsupervised manner (see Fig. 1(a)), which has been proven to be prohibitively difficult and even infeasible [50]. Therefore, most prior works (see Fig. 1(b)) aim to learn domain-invariant representations through elaborate feature alignment modules, such as adversarial training [21] and pseudo-labeling [37], without exploring the intrinsic composition of the features themselves. Despite their general efficacy, these methods may introduce a significant amount of redundant and noisy information during the alignment process. For humans, when identifying an object, we tend to first look for its main components, which can be regarded as a process of information compression from a learning perspective. Inspired by this, we aim to mimic the very first step of the human recognition process by identifying the main components from object images without human prior knowledge or fine-grained annotations.

To be specific, we introduce a self-supervised reconstruction module based on Slot Attention [51], termed Selective Slot-based Reconstruction (SSR) , to assist deep models in learning a set of part-based representations for sparsely characterizing a target object. Formally, given an input x , g extracts a set of feature embeddings z ∈ R N × d z , where N is the number of embedding and d z denotes embedding dimension. The Slot Attention module will first take a set of slots s ∈ R K × d s ( K is the number of slots) and the feature map z , then project them to dimension d s by a linear transformation f k for slots

and f q , f v for z . The Slot Attention will be trained as follows,

where q = f q ( z ) ∈ R K × d s , k = f k ( z ) ∈ R N × d s , and v = f v ( z ) ∈ R N × d s denote the query, key and value vectors respectively. A ∈ R N × K stands for the attention matrix. Unlike self-attention, the queries in slot attention are a function of the slots s ∼ N ( s ; µ , σ ) ∈ R K × d s , and will be updated iteratively over T iterations. The slots are initialized by random sampling from a standard Gaussian distribution. Queries at iteration t are represented by ˆ q t = f q ( s t ) , and the slot updating process is: s t +1 := update ( attn (ˆ q t , k ) , v ) . After each iteration, a Gated Recurrent Unit (GRU) is employed on the slot representations s t +1 to update their states, integrating new information while retaining relevant context from previous iterations.

Since the number of slots K is manually predefined and fixed during training, attention networks may learn redundant or even incorrect associations, thus affecting the understanding of objects and scenes. Therefore, our SSR introduces a selective slot attention mechanism that dynamically adjusts the importance of slots for reconstruction. To quantitatively represent which slots are more important for reconstruction, we first introduce an importance score ρ for each slot. For each slot s i , we use a lightweight neural network h θ to predict an initial importance score: ρ init i = σ ( h θ ( s i )) , where σ ( · ) is the sigmoid function, ensuring ρ init i ∈ [0 , 1] . To capture interactions among slots, we further introduce an interaction matrix W ∈ R K × K , where each element W ij is defined as W ij = softmax j ( ( U s i ) ⊤ ( V s j ) / √ d ) . Here, U and V are learnable projection matrices, and d is a scaling factor. The final importance score ρ i for each slot s i is then computed as ρ i = ∑ K j =1 W ij ρ init j . We then scale each slot representation by its respective importance score to create a weighted slot representation ˜ s i = ρ i · s i . To this end, the overall training objective of SSR can be formulated as:

where λ is the balancing parameter. The regularization term encourages sparsity in the initial importance scores, guiding the model to utilize only the most relevant slots for reconstruction. On the other hand, the less relevant slots are down-weighted rather than discarded, which may help retain subtle information. Compared to continuous semantic representations, this training process is more manageable due to the discretization of the slots. Since we lack truly fine-grained annotations, the learned slots or 'concepts' may not be as readily interpretable as individual words. It is worth noting that incorporating a strong visual feature extractor could significantly boost baseline performance. However, for the sake of fairness, we have opted not to use approaches, such as DINOSAUR [64].

3.2 High-Order Relational Reasoning (HORR)

Albeit we have obtained part-based representations in the form of slots, their inner- and interobject relations are still under-explored. Prior graph-based [9, 7] and non-graph [33] works only accommodate pairwise relationships ( e.g., connection between two nodes in a simple graph), but are inadequate to model the high-order relations inherent in images. Addressing this issue, we introduce a simple yet effective technique-HORR-to model high-order topological relations with a collection of graph nodes and hyperedges, divided into three stages: construction, learning, and matching.

Definition 1 (Topological Homogeneity) We conceptualize topological homogeneity between the same object across images as a (hyper)graph matching problem, which is mathematically formulated as a relaxed quadratic assignment problem [52],

where A ∈ R n × n and B ∈ R m × m are the adjacent matrix encoding structure information of the graph G A and G B respectively, n and m are the number of graph nodes, || · || F is the Frobenius norm, X u ∈ R m × n is the unary affinity matrix and generally specified as the node affinity M aff , and X is the relaxed permutation matrix encoding node-to-node assignment.

Hypergraph Construction. Given visual slot representations from the previous step, we carry out a linear transformation to obtain graph nodes V A and V B . This transformation is intended to map the visual features into the graph domain, ensuring effective graph matching. Then, we build hypergraph graphs G A and G B in both domains, modeling the topological structures inherent in the images, x A and x B , respectively. The hypergraph [17] is defined as: G A/B = ( V A/B , E A/B , W ) , where V A/B is the node sets, E A/B is the hyperedge sets, and W ∈ R |E|×|E| is a diagonal matrix of edge weights. The hypergraph is denoted by an incidence matrix H ∈ R |V|×|E| , where H ( v, e ) = 1 indicates the node v ∈ e and H ( v, e ) = 0 indicates v / ∈ e . We adopt non-parametric density estimation, i.e., K-Nearest Neighbors (KNN), to establish hyperedge connections. The Euclidean distance is used to calculate the distance between node embeddings.

Hypergraph Learning. After constructing the hypergraphs, we update the graph node features according to the connectivity defined by the hyperedges. Technically, we introduce hypergraph convolution [17] to perform message-passing and feature aggregation,

where Θ represents the parameter to be learned during training. By doing so, we can explicitly model intricate correlations among different parts of an image ( i.e., slots). On the other hand, in contrast to grids, sequences, and even graphs, this high-order modeling can more effectively capture relations among nodes, avoiding excessive or erroneous pairwise connections.

Hypergraph Matching. Since the graph node features have been updated by Eq. (5), we introduce an affinity matrix W aff to measure the node correspondence between G A and G B . Following [19, 44], we use the differentiable Sinkhorn layer [67] to calculate the affinity matrix ˆ W aff , where each element indicates the degree of matching between pairs of nodes across graphs. To leverage the topological structures, we need a training objective to minimize the pairwise structural discrepancy between the hypergraphs [78]. In particular, we follow the Definition 1 to specify unary affinity matrix X u as the obtained node affinity matrix ˆ W aff . Since the constructed hypergraphs naturally encode rich high-order relationships, they enhance the cross-domain topological matching process. Conversely, the matching process introduces additional structural knowledge to the current graph through message propagation. Formally, the hypergraph-based matching objective is formulated as follows:

where the ( i, j ) element in Y Π ∈ R n × m is 1 if v A i ∈ G A and v B j ∈ G B belong to the same object class, otherwise it will be 0 . Here, Y Π can be regarded as pseudo labels for matching due to the absence of ground-truth correspondence. Moreover, as the matching primarily targets high-level semantic relationships, we avoid imposing additional structural constraints [23, 43, 44] on this training objective, thereby significantly simplifying the training process.

3.3 Training and Inference

Putting everything together, the full training objective is formulated as follows:

where α and β are hyper-parameters for balancing different loss terms. Our algorithm first trains deep models using the reconstruction objective to obtain discrete part representations with sufficient sparsity. Then it turns to model the cross-image components correlations as the graph matching problem where we further introduce the structural regularization term into the matching loss.

Note that our approach can be directly applied to test-time adaptation, similar to those self-supervised methods [69, 49, 3, 20]. The difference lies in the fact that their proxy tasks are somewhat heuristic, such as predicting rotations, jigsaw puzzles, and random masking. In contrast, our method directly exploits the intrinsic structured properties of the data, making it more robust and versatile.

4 Experiments

In this section, we empirically evaluate the proposed REMA in two types of OOD scenarios, i.e., OOD generalization and test-time adaptation. In the following, we first describe the experimental setup (Section 4.1) and then provide the main results (Section 4.2) and ablation studies (Section 4.3).

4.1 Experimental Setup

Datasets. For OOD generalization, we leverage the three most widely used benchmark datasets. PACS [39] comprises 9,991 images and exhibits significant variations in image styles. It consists of 4 domains each with 7 classes, i.e., Photo, Art Painting, Cartoon, Sketch . VLCS contains 10,729 images of 5 classes from 4 photographic domains: PASCAL VOC 2007, LabelMe, Caltech, Sun . Office-Home [73] is collected from both office and home environments, and its domain shifts stem from variations in viewpoint and image style. It has 15,500 images of 65 classes from 4 domains, i.e., Artistic, Clipart, Product, Real World . Regarding test-time adaptation, we follow the common benchmarks [75, 30, 79] that utilize CIFAR-10/100 [35] and ImageNet [14] as the ID (training) data. CIFAR-10/100C [26] and Imagenet-C [26] are used as OOD (test) data, comprising different corruptions applied to their original datasets.

Implementation Details. For OOD generalization, we use ResNet-50 for PACS, Office-Home, and VLCS and ResNet-18 for CIFAR-10. The model is trained using stochastic gradient descent with momentum 0.9, and weight decay 10 -4 . The training batch size is set to 128. The learning rate is 10 -4 . Following common practice, the model selection is based on a training domain validation set. For test-time adaptation, we use ResNet-50 for all datasets. We utilize the Adam optimizer to update the network parameters. To facilitate a fair comparison, the test batch size of 64 in all methods. λ in Eq. (3) is set to 0.01 in all experiments. α and β in Eq. (7) is set to 10 and 0.1, respectively.

Baselines. We compare REMA against two types of baseline methods. (1) OOD Generalization: We adopt the leave-one-domain-out evaluation protocol and follow the model selection strategy used in [25]. The main baselines have optimization-based CORAL [68], MLDG [40], CDANN [45] and MMD [42], augmentation-based Mixstyle [95], SagNet [55] and I-Mixup [80, 82, 84], and so on. (2) Test-time adaptation: All methods are based on online batch-level test data adaptation setting and the following baselines are included: entropy-minimization: Tent [75] and SHOT [47], pseudo-labeling: T3A [29] and TAST [31], consistency-alignment: TSD [79] and TIPI [56].

Impulse Noise

Defocus Blur

Snow

Frost

Fog

Brightness

Contrast

Elastic Transform

Pixelate

Jpeg Compression

65

70

75

80

85

Acc

ERM

Ours

4.2 Main Results

OOD Generalization. The main results are presented in Table 1 and Figure 3. Notably, REMA consistently outperforms all baseline methods by a large margin in each dataset. For example, compared to the recent method VNE, REMA improves classification accuracy by 1.8% for PACS, 6.1% for Office-Home, and 1.3% for VLCS, revealing the importance of our reconstruction (semantic abstraction) and matching (topological homogeneity) modules. Moreover, there are two notable observations: (1) Compared to statistical matching methods ( e.g., CORAL, DANN, and MMD) that directly optimize moment matching objectives in the latent space, REMA demonstrates superior performance by introducing topological information with more sparse input (slots). (2) Style or data augmentation based methods ( e.g., MixStyle) is simple and easy-to-implement. However, their intuitive nature of interpolating around the training set may not accurately cover the target distribution region. By contrast, REMA directly learns major components from training data. (3) Compared to PACS and VLCS datasets, the Office-Home dataset has more categories and total samples. Many baseline approaches even perform worse than ERM, indicating their limited scalability. Instead, REMA demonstrates a larger performance gain in this challenging task.

Test-Time Adaptation. We compared REMA with existing advanced TTA methods, and the results are summarized in Table 2 and Table 3. REMA consistently provides improvements on multiple datasets. Compared to TSD [79], it achieves a 1.3% improvement on three pixel-level corruption datasets and a 3.1% improvement on three challenging cross-domain datasets. Compared to the SOTA, REMA has three major advantages: (1) Tent [75] or SHOT [47] based on entropy minimization has limitations, as they can overcome distribution shifts caused by pixel corruption but fail to handle cross-domain test data, even leading to performance degradation. However, REMA can handle both types of OOD data. (2) T3A [29] and TAST [31] based on pseudo-labels have performance limitations, and the pros and cons of pseudo-labels significantly impact the adaptation, which can be

Method

VLCS

PACS

Office-Home

Average Acc. (%)

ERM

76.7

±

0

.

5

83.2

±

1

.

1

67.1

±

1

.

0

75.3

+Tent [75]

73.0

±

1

.

3

85.2

±

0

.

6

66.3

±

0

.

8

74.9

+TentClf [75]

75.8

±

0

.

7

82.7

±

1

.

6

66.8

±

1

.

0

75.1

+SHOT [47]

67.1

±

0

.

9

84.1

±

1

.

2

67.6

±

0

.

7

72.9

+T3A [29]

77.3

±

0

.

4

83.9

±

1

.

1

68.3

±

0

.

8

76.5

+TAST [31]

77.7

±

0

.

5

84.1

±

1

.

2

68.6

±

0

.

7

76.8

+TAST-BN [31]

73.5

±

1

.

4

89.2

±

0

.

5

68.9

±

0

.

5

77.2

+TSD [79]

74.5

±

0

.

9

89.3

±

0

.

6

68.4

±

0

.

7

77.3

+REMA

79.4

±

0

.

4

90.3

±

0

.

3

71.6

±

0

.

6

80.4

w/o REMA

w/ REMA

Test Data

Impulse Noise

Defocus Blur

Snow

Frost

Fog

Brightness

Contrast

Elastic Transform

Pixelate

Jpeg Compression

40

50

60

70

80

90

Acc

ERM

Tent

EATA

DeYO

Ours

(a) Grad-CAM Visualization

(b) Adaptation Order (

→

)

observed in their ceilings on two types of datasets. REMA enhances resistance to noisy labels by capturing higher-order semantic dependencies. (3) REMA converts dense pixels into sparse slots, accurately capturing domain-invariant features, which will not be affected by pixel corruption and domain shift. Implementing consistent optimization during test time makes REMA more robust, resulting in a 5.1% accuracy improvement across all benchmarks, i.e., ERM vs. REMA.

4.3 Ablation Studies

Ablations of key modules in REMA. In this part, we provide the ablation results in Table 4, investigating the independent and combined effects of SSR and HORR proposed in REMA. As can be seen, incorporating SSR and HORR separately leads to improved generalization performance, demonstrating their contributions to abstraction and relational modeling. In addition,

integrating SSR and HORR in our method yields the best performance, highlighting that the two modules systematically work together and reciprocate each other. Moreover, as shown in Figure 4(a), the full REMA can provide more complete and accurate representations of objects.

Analysis on continuous test-time adaptation. We empirically evaluate the continuous learning ability of REMA by comparing it with state-of-the-art test-time adaptation methods, namely Tent [75], EATA [57], and DeYO [38]. The results are presented in Fig. 4(b), where the adaptation order is from left to right. We can see that the proposed REMA substantially and consistently outperforms all baseline methods as adaptation proceeds, revealing the robustness and the capability to handle open-world changes.

Analysis on hypergraph matching. In Figure 5, we visualize the learned doubly stochastic node affinity matrix and the ground-truth (GT) matrix. We observe that the proposed graph matching method effectively identifies the correct node affinity,

Predicted Results

Ground Truth

revealing its effectiveness in handling cross-domain higher-order relationships.

(a)

ERM

(b)

REMA w/o SSR

(c)

REMA w/o HORR

(d)

REMA (Full)

airplane

automobile

puq

cat

dear

dog

frog

horse

ship

truck

Feature visualization. Figure 6 demonstrates the t -SNE [71] visualization of feature embeddings for ERM, REMA w/o SSR, REMA w/o HORR, and REMA (Full). We extract feature embeddings using CIFAR-10C data, where the corruption type is snow with a severity level of 5. The different color stands for different classes. We can observe that our REMA exhibits better clustering patterns with both SSR and HORR modules. When SSR is removed, some closely related categories ( e.g., animals) become highly mixed, indicating the advantage of SSR in extracting robust semantic representations. On the other hand, without HORR, intra-class variations increase, indicating some degree of ambiguity between different categories. Our HORR addresses semantic ambiguity by modeling topological homogeneity.

5 Related Work

OOD Generalization. The aim of OOD Generalization is to train a model using data from the source distribution so that it can perform well on an unknown target distribution. A series of works can be divided into three categories: (1) Minimizing distribution differences [42, 45, 68, 90]: using meta-learning to simulate distribution shift [40, 18], or learning an invariant transformation based on adversarial learning [42, 45, 89, 94, 93]. (2) Domain-invariant representation learning: by analyzing the feature learning process of deep neural networks [16, 66, 27, 63, 12, 13], it is inferred how ID data can be generalized to OOD data [6, 65]. By understanding the interactions between ERM and generalization, domain-invariant features are learned. (3) More generic OOD generalization: many studies focus on generalization under a relaxed assumption, such as open environments with unknown classes [8], semi-supervised settings in the testing environment [86], adapting foundation models [91], and OOD data synthesis [60, 59, 22], aiming to identify generalizable features from sophisticated test distributions. However, previous studies have only focused on the instances themselves and have not explored the inherent high-order semantic dependencies in ID and OOD data, overlooking the topological homogeneity between distributions.

Test-time Adaptation. The goal of TTA [46] is to adjust the source trained model using test data without ground truth during the testing phase [47, 58, 10, 87, 79, 29]. Most of the existing methods directly perform gradient optimization on the test samples, with optimization objectives including prediction entropy [75], self-training and stochastic restoring [77], or based on normalization [54], etc., making the model adapt to the dynamic target environment at each step. Recently works further consider the scenario of noisy outlier samples [24, 70] appearing in the test stream. However, previous studies have simply optimized the test instances based on self-training, ignoring the topological relationship between samples and the high-order semantic dependencies within the test data steam, leading to suboptimal adaptation.

6 Conclusion

In this paper, we propose to achieve out-of-distribution robustness from the perspective of ensuring topological homogeneity between the same object class regardless of the surrounding environments. To achieve this goal, we propose a new framework REMA to first obtain a set of sparse and discrete representations from dense pixels and then model the high-order topological relations and dependencies via hypergraph (a generalized form of graph). Our experimental results reveal that REMA achieves superior performance on standard OOD generalization and test-time adaptation benchmarks. In the future, we aim to extend this framework to more complex and dynamically changing scenes.

References

Kartik Ahuja, Karthikeyan Shanmugam, Kush Varshney, and Amit Dhurandhar. Invariant risk minimization games. In ICML , pages 145-155, 2020.

Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.02893 , 2019.

Alexander Bartler, Andre Bühler, Felix Wiewel, Mario Döbler, and Bin Yang. Mt3: Meta test-time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics , pages 3080-3090. PMLR, 2022.

Irving Biederman. Recognition-by-components: a theory of human image understanding. Psychological review , 94(2):115, 1987.

Gilles Blanchard, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. The Journal of Machine Learning Research , 22(1):46-100, 2021.

Yuan Cao, Zixiang Chen, Misha Belkin, and Quanquan Gu. Benign overfitting in two-layer convolutional neural networks. Advances in neural information processing systems , 35:25237-25250, 2022.

Chaoqi Chen, Jiongcheng Li, Hong-Yu Zhou, Xiaoguang Han, Yue Huang, Xinghao Ding, and Yizhou Yu. Relation matters: Foreground-aware graph-based relational reasoning for domain adaptive object detection. TPAMI , 2022.

Chaoqi Chen, Luyao Tang, Yue Huang, Xiaoguang Han, and Yizhou Yu. Coda: Generalizing to open and unseen domains with compaction and disambiguation. NeurIPS , 36, 2023.

Chaoqi Chen, Luyao Tang, Feng Liu, Gangming Zhao, Yue Huang, and Yizhou Yu. Mix and reason: Reasoning over semantic topology with data mixing for domain generalization. NeurIPS , 35:33302-33315, 2022.

Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR , pages 295-305, 2022.

Liang Chen, Yong Zhang, Yibing Song, Ying Shan, and Lingqiao Liu. Improved test-time adaptation for domain generalization. In CVPR , 2023.

Liang Chen, Yong Zhang, Yibing Song, Anton Van Den Hengel, and Lingqiao Liu. Domain generalization via rationale invariance. In ICCV , pages 1751-1760, 2023.

Liang Chen, Yong Zhang, Yibing Song, Zhen Zhang, and Lingqiao Liu. A causal inspired early-branching structure for domain generalization. IJCV , pages 1-21, 2024.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on , pages 248-255. Ieee, 2009.

Abhimanyu Dubey, Vignesh Ramanathan, Alex Pentland, and Dhruv Mahajan. Adaptive methods for real-world domain generalization. In CVPR , 2021.

Nelson Elhage, Tristan Hume, Catherine Olsson, Nicholas Schiefer, Tom Henighan, Shauna Kravec, Zac Hatfield-Dodds, Robert Lasenby, Dawn Drain, Carol Chen, et al. Toy models of superposition. arXiv preprint arXiv:2209.10652 , 2022.

Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. In AAAI , volume 33, pages 3558-3565, 2019.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In ICML , pages 1126-1135, 2017.

Kexue Fu, Shaolei Liu, Xiaoyuan Luo, and Manning Wang. Robust point cloud registration framework based on deep graph matching. In CVPR , pages 8893-8902, 2021.

Yossi Gandelsman, Yu Sun, Xinlei Chen, and Alexei Efros. Test-time training with masked autoencoders. NeurIPS , 35:29374-29385, 2022.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR , 17(1):2096-2030, 2016.

Jiahui Gao, Renjie Pi, Yong Lin, Hang Xu, Jiacheng Ye, Zhiyong Wu, Weizhong Zhang, Xiaodan Liang, Zhenguo Li, and Lingpeng Kong. Self-guided noise-free data generation for efficient zero-shot learning, 2023.

Quankai Gao, Fudong Wang, Nan Xue, Jin-Gang Yu, and Gui-Song Xia. Deep graph matching under quadratic constraint. In CVPR , pages 5069-5078, 2021.

Taesik Gong, Yewon Kim, Taeckyung Lee, Sorn Chottananurak, and Sung-Ju Lee. Sotta: Robust test-time adaptation on noisy data streams. Advances in Neural Information Processing Systems , 36, 2024.

Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR , 2021.

Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations , 2019.

Katherine Hermann and Andrew Lampinen. What shapes feature representations? exploring datasets, architectures, and training. Advances in Neural Information Processing Systems , 33:9995-10006, 2020.

Zeyi Huang, Haohan Wang, Eric P Xing, and Dong Huang. Self-challenging improves cross-domain generalization. In ECCV , pages 124-140, 2020.

Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. NeurIPS , 34:2427-2440, 2021.

Minguk Jang and Sae-Young Chung. Test-time adaptation via self-training with nearest neighbor information. In ICLR , 2023.

Minguk Jang, Sae-Young Chung, and Hye Won Chung. Test-time adaptation via self-training with nearest neighbor information. In The Eleventh International Conference on Learning Representations , 2023.

Daehee Kim, Youngjun Yoo, Seunghyun Park, Jinkyu Kim, and Jaekoo Lee. Selfreg: Self-supervised contrastive regularization for domain generalization. In ICCV , pages 9619-9628, 2021.

Jaeill Kim, Suhyun Kang, Duhun Hwang, Jungwook Shin, and Wonjong Rhee. Vne: An effective method for improving deep representation by manipulating eigenvalue distribution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3799-3810, 2023.

Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In ICML , pages 5637-5664, 2021.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, University of Toronto , 2009.

David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapolation. In International Conference on Machine Learning , pages 5815-5826, 2021.

Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on Challenges in Representation Learning, ICML , volume 3, page 2, 2013.

Jonghyun Lee, Dahuin Jung, Saehyung Lee, Junsung Park, Juhyeon Shin, Uiwon Hwang, and Sungroh Yoon. Entropy is not enough for test-time adaptation: From the perspective of disentangled factors. In ICLR , 2024.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. In ICCV , pages 5542-5550, 2017.

Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta-learning for domain generalization. In AAAI , 2018.

Da Li, Jianshu Zhang, Yongxin Yang, Cong Liu, Yi-Zhe Song, and Timothy M Hospedales. Episodic training for domain generalization. In ICCV , pages 1446-1455, 2019.

Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In CVPR , pages 5400-5409, 2018.

Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma: Semantic-complete graph matching for domain adaptive object detection. In CVPR , 2022.

Wuyang Li, Xinyu Liu, and Yixuan Yuan. Sigma++: Improved semantic-complete graph matching for domain adaptive object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(7):9022-9040, 2023.

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In ECCV , pages 624-639, 2018.

Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribution shifts. arXiv preprint arXiv:2303.15361 , 2023.

Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In ICML , pages 6028-6039. PMLR, 2020.

Chang Liu, Xinwei Sun, Jindong Wang, Haoyue Tang, Tao Li, Tao Qin, Wei Chen, and Tie-Yan Liu. Learning causal semantic representation for out-of-distribution prediction. NeurIPS , 34, 2021.

Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Ttt++: When does self-supervised test-time training fail or thrive? NeurIPS , 34:21808-21820, 2021.

Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Schölkopf, and Olivier Bachem. Challenging common assumptions in the unsupervised learning of disentangled representations. In ICML , pages 4114-4124, 2019.

Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. NeurIPS , 33:11525-11538, 2020.

Eliane Maria Loiola, Nair Maria Maia De Abreu, Paulo Oswaldo Boaventura-Netto, Peter Hahn, and Tania Querido. A survey for the quadratic assignment problem. European journal of operational research , 176(2):657-690, 2007.

Seonwoo Min, Nokyung Park, Siwon Kim, Seunghyun Park, and Jinkyu Kim. Grounding visual representations with texts for domain generalization. In European Conference on Computer Vision , pages 37-53. Springer, 2022.

Zachary Nado, Shreyas Padhy, D Sculley, Alexander D'Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006.10963 , 2020.

Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. In CVPR , pages 8690-8699, 2021.

A Tuan Nguyen, Thanh Nguyen-Tang, Ser-Nam Lim, and Philip HS Torr. Tipi: Test time adaptation with transformation invariance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 24162-24171, 2023.

Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In ICML , pages 16888-16905. PMLR, 2022.

Prashant Pandey, Mrigank Raman, Sumanth Varambally, and Prathosh Ap. Generalization on unseen domains via inference-time label-preserving target projections. In CVPR , pages 12924-12933, 2021.

Renjie Pi, Tianyang Han, Wei Xiong, Jipeng Zhang, Runtao Liu, Rui Pan, and Tong Zhang. Strengthening multimodal large language model with bootstrapped preference optimization, 2024.

Renjie Pi, Jianshu Zhang, Jipeng Zhang, Rui Pan, Zhekai Chen, and Tong Zhang. Image textualization: An automatic framework for creating accurate and detailed image descriptions, 2024.

Vihari Piratla, Praneeth Netrapalli, and Sunita Sarawagi. Efficient domain generalization via commonspecific low-rank decomposition. In ICML , pages 7728-7738, 2020.

Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations , 2020.

Wojciech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Klaus-Robert Müller. Explainable AI: interpreting, explaining and visualizing deep learning , volume 11700. Springer Nature, 2019.

Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann SimonGabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap to real-world object-centric learning. arXiv preprint arXiv:2209.14860 , 2022.

Ruoqi Shen, Sébastien Bubeck, and Suriya Gunasekar. Data augmentation as feature manipulation. In International conference on machine learning , pages 19773-19808. PMLR, 2022.

Ravid Shwartz-Ziv and Naftali Tishby. Opening the black box of deep neural networks via information. arXiv preprint arXiv:1703.00810 , 2017.

Richard Sinkhorn. A relationship between arbitrary positive matrices and doubly stochastic matrices. The annals of mathematical statistics , 35(2):876-879, 1964.

Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV , pages 443-450, 2016.

Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In ICML , pages 9229-9248. PMLR, 2020.

Devavrat Tomar, Guillaume Vray, Jean-Philippe Thiran, and Behzad Bozorgtabar. Un-mixing test-time normalization statistics: Combatting label temporal correlation. arXiv preprint arXiv:2401.08328 , 2024.

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR , 9(11), 2008.

Vladimir N Vapnik. An overview of statistical learning theory. IEEE transactions on neural networks , 10(5):988-999, 1999.

Hemanth Venkateswara, Jose Eusebio, Shayok Chakraborty, and Sethuraman Panchanathan. Deep hashing network for unsupervised domain adaptation. In CVPR , pages 5018-5027, 2017.

Riccardo Volpi, Hongseok Namkoong, Ozan Sener, John C Duchi, Vittorio Murino, and Silvio Savarese. Generalizing to unseen domains via adversarial data augmentation. In NeurIPS , pages 5334-5344, 2018.

Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In ICLR , 2021.

Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey on domain generalization. TKDE , 2022.

Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7201-7211, 2022.

Runzhong Wang, Junchi Yan, and Xiaokang Yang. Neural graph matching network: Learning lawler's quadratic assignment problem with extension to hypergraph and multiple-graph matching. IEEE TPAMI , 44(9):5261-5279, 2021.

Shuai Wang, Daoan Zhang, Zipei Yan, Jianguo Zhang, and Rui Li. Feature alignment and uniformity for test time adaptation. In CVPR , 2023.

Yufei Wang, Haoliang Li, and Alex C Kot. Heterogeneous domain generalization via domain mixup. In IEEE International Conference on Acoustics, Speech and Signal Processing , pages 3622-3626, 2020.

Zehao Xiao, Xiantong Zhen, Shengcai Liao, and Cees GM Snoek. Energy-based test sample adaptation for domain generalization. In ICLR , 2023.

Minghao Xu, Jian Zhang, Bingbing Ni, Teng Li, Chengjie Wang, Qi Tian, and Wenjun Zhang. Adversarial domain adaptation with domain mixup. In AAAI Conference on Artificial Intelligence , pages 6502-6509, 2020.

Zhenlin Xu, Deyi Liu, Junlin Yang, Colin Raffel, and Marc Niethammer. Robust and generalizable visual representation learning via random convolutions. In ICLR , 2021.

Shen Yan, Huan Song, Nanxiang Li, Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.00677 , 2020.

Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain generalization. In CVPR , pages 8024-8034, 2022.

Lei Zhang, Ji-Fu Li, and Wei Wang. Semi-supervised domain generalization with known and unknown classes. NeurIPS , 36, 2023.

Marvin Mengxin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. In NeurIPS , 2022.

Marvin Mengxin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. In Advances in Neural Information Processing Systems , 2021.

Yabin Zhang, Bin Deng, Hui Tang, Lei Zhang, and Kui Jia. Unsupervised multi-class domain adaptation: Theory, algorithms, and practice. IEEE TPAMI , 2020.

Yabin Zhang, Minghan Li, Ruihuang Li, Kui Jia, and Lei Zhang. Exact feature distribution matching for arbitrary style transfer and domain generalization. In CVPR , 2022.

Yabin Zhang, Wenjie Zhu, Hui Tang, Zhiyuan Ma, Kaiyang Zhou, and Lei Zhang. Dual memory networks: A versatile adaptation approach for vision-language models. In CVPR , 2024.

Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey. TPAMI , 2022.

Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Deep domain-adversarial image generation for domain generalisation. In AAAI , pages 13025-13032, 2020.

Kaiyang Zhou, Yongxin Yang, Timothy Hospedales, and Tao Xiang. Learning to generate novel domains for domain generalization. In ECCV , pages 561-578, 2020.

Kaiyang Zhou, Yongxin Yang, Yu Qiao, and Tao Xiang. Domain generalization with mixstyle. In ICLR , 2021.

Xiao Zhou, Yong Lin, Weizhong Zhang, and Tong Zhang. Sparse invariant risk minimization. In ICML , pages 27222-27244, 2022.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims in the abstract and introduction accurately represent the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We provide this part in the supplementary document.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All assumptions and definitions have been clearly stated.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We have fully disclosed all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: After internal review and patent approval, we will release the code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We have provided all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We have provided the standard deviation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We have provided these details in the supplementary.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have read the Code of Ethics and will always adhere to the principles it requires.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have provided this part in the supplementary.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use publicly available datasets and code, and we have properly cited their papers or project addresses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: [NA]

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method for improving domain generalization and test time adaptation by introducing a selective variant of slot attention, formulating the relationship between slots across images as topological homogeneity between hypergraphs constructed based on the slots, and thereby matching occurances of the same object between different images.\n\nSTRENGTHS:\nThe proposed method and its components introduce novel techniques for the domain generalization problem that are well motivated and appear to be sound.\n\nThe mathematical derivation of the employed forms of the individual steps makes sense.\n\nSeveral ablations well demonstrate the dependence of REMA's results on its main building blocks.\n\nWEAKNESSES:\nThe proposed method consists of fairly many newly introduced steps.\nWhile ablations show their necessity of the major modular building blocks of REMA for achieving the eventual results, these building blocks are not analyzed in detail.\nFor understanding them (and therefore REMA), it would be necessary to expound each individual step and confirm that it has the intended effect and that the explicit or implicit mathematical assumptions hold.\nFor example, the results of standard slot attention can be of varying quality, and it would be important to verify whether and how such imprecisions introduced there propagate through the next steps.\nThe analysis in Figure 5 goes in this direction, but it is not clear what exactly is probed and measured here. A similar analysis as in for the other components of the method would be helpful.\n\n**Main issue of the paper**: There exist different communities in ML which study the generalization performance of DNNs to distribution shifts in very different settings. Unsupervised Domain Adaptation (UDA) is separate from Domain Generalization (DG) and both are separate from OOD generalization. Putting papers which study either of those in the same Table is wrong, misleading and highly confusing. All of these settings have their benefits and challenges and the numbers are simply incomparable if the initial conditions, such as which data is available during training, differ this strongly. The authors conflate and confuse all three settings on multiple occasions which makes the paper confusing and the results are impossible to understand. It is further completely unclear in which of the three their proposed method falls into. Below, I explain the issue in much more detail and provide specific problematic text instances. The paper needs a major revision which includes reworking all sections and most Tables.\n\nline 183: \"For OOD generalization, we leverage the three most widely used benchmark datasets.\" The datasets PACS and Office-Home are usually used in Unsupervised Domain Adaptation (UDA), not OOD generalization. Datasets commonly used to benchmark OOD generalization are datasets like ImageNet-R, ImageNet-Sketch, ObjectNet or ImageNet-V2. The choice of the baselines the authors compare to also gives the impression that they confuse domain adaptation with OOD generalization: CORAL and DANN are UDA methods.\n\n\nline 66: \"The goal of OOD generalization is to find a predictor f : X 7→ Y that generalizes well to all unseen target domains.\" This is correct, but then the used baselines are wrong because they assume access to the target domains and are trained using this information. It is not clear to me how the UDA methods (shown in Table 1) can be implemented without this access.\n\n\nThe setting of UDA differs drastically from OOD generalization: In UDA, we attempt to learn a model which performs well on the source domain and a set of target domains, using unlabeled data from the target domains. That is, in UDA, we assume access to the unlabeled test set at training time. This differs from the OOD generalization setting: Here, we want to train a model on a source dataset and then test its generalization to an unseen test set. That is, we assume no knowledge about the test time distribution shift. This is a crucial difference and it is wrong to confuse the two terms. The implemented baselines, e.g. CORAL and DANN, are used in a UDA setup, not in an OOD generalization setup. In this light, the wording \"Following common practice, the model selection is based on a training domain validation set\" is meaningless because the common practice for model selection differs depending on which setting we consider.\n\n\nThe related work section is similarly confusing. On the one hand, the authors wish to review works on OOD generalization where the goal is \"to train a model using data from the source distribution so that it can perform well on an unknown target distribution\". [41] and [42] use the DG setting, i.e. assumes access to n-1 target domains. [63] assumes access to all target domains. It is unclear why [61] is a good citation for domain-invariant learning. I haven't checked all citations, but the ones I checked **all** assumed access to the target domains.\n\n\nThe setting the baselines use in Table 1 is inconsistent. CORAL and DANN have been trained in an UDA setting. That is, they assume access to unlabeled target data at test time from a target domain. VNE and IRM are used in a domain generalization setting where they assume access to n-1 domains and aim to perform well on an unseen domain. These are two very different settings which are incomparable to each other and it is wrong and confusing to put them in the same table without any discussion. Further, the authors aim to test OOD generalization where no access to unseen target domain data is available (line 66) which is incompatible with either UDA or domain generalization.\n\n\nThe baseline numbers for the cited papers are wrong. I checked the numbers of CORAL and VNE from Table 1 with the numbers in the original papers and they do not match. If the authors reimplemented all the baselines, they need to state this and discuss where the performance differences come from.\n\n\nChecking papers-with-code for the PACS dataset (https://paperswithcode.com/sota/domain-generalization-on-pacs-2), the best numbers are close to 100% for the domain generalization setting. Since the authors use DG benchmarks in their table 1, I assume that using other benchmarks in the DG domain would be valid as well. If we filter for RN50 architectures only, the best number is 90.5% which is higher compared to the best number reported by the authors.\n\n\nLooking at the Office-Home dataset (https://paperswithcode.com/sota/unsupervised-domain-adaptation-on-office-home), the best number here is 90%. Here, the setting is UDA. But as mentioned before, it is not clear which setting should be used because the authors include both UDA and DG benchmarks in their Table 1. The best number in the UDA setting is higher than what the authors report in Table 1. There are actually five separate benchmarks for OfficeHome on papers with code (Domain Generalization, Domain Adaptation, Universal Domain Adaptation, UDA, Partial DA) and it is not clear which one should be used.\n\n\nIt is not clear how the models are trained. The test sets are comprised of different domains. Do the authors train their models on all of them or on e.g. two of the domains and test against the rest or on one of the domains and then test against the rest?\n\n\nFor test-time adaptation, the authors missed important works which perform better than their method. SLR achieves an error of 48.7% on the highest severity of ImageNet-C [A]. ETA and EATA [B] achieve an error of around 52%. \n[A] Mummadi et al. \"Test-time adaptation to distribution shift by confidence maximization and input transformation\"\n[B] Niu et al. \"Efficient test-time model adaptation without forgetting.\"\n\n\nPress et al. [C] showed that most test-time adaptation methods collapse when adapting for long periods of time. The authors should test their model on the proposed CCC benchmark to analyze whether their method also suffers from the collapse. The authors are writing that they test their method in  the continuous test-time adaptation scenario in Fig. 4b, but it is entirely unclear on which dataset they are testing it. Is it the CCC benchmark? Notably, if the authors are testing their approach on the Continual Test-time adaptation benchmark from [D] (although I am just guessing here), Press et al. showed in [C] that one needs to adapt for longer time periods to show-case collapse and the adaptation periods proposed in [D] are insufficient. The authors must state clearly which continual learning benchmark they are using and I would suggest to report numbers on both [C] and [D].\n[C] Press et al. \"RDumb: A simple approach that questions our progress in continual test-time adaptation\"\n[D] Wang et al. \"Continual test-time domain adaptation\"\n\nQUESTIONS:\nThe paper requires a major revision to make it much more clear which setting the authors are targetting: Is it:\n- OOD generalization as suggested by the title? Then, the authors **cannot** assume any knowledge about the distribution shift at test time. Then the baselines in Table 1 cannot be used as they do assume access to the target domains.\n- Unsupervised Domain Adaptation as suggested by the baselines in Table 1, i.e. DANN and CORAL? Then, the authors need to reformulate their title, abstract, motivation etc. and remove all instances of \"OOD generalization\" because this is very confusing otherwise. The authors also need to remove the Domain Generalization baselines from Table 1.\n- Domain Generalization as suggested by the IRM and VNE baselines in Table 1? Then, the authors need to remove the UDA baselines from their Table 1 and remove instances of mentioning OOD generalization from the paper.\n\n\nThe authors need to add Test-Time adaptation baselines to their Table and improve their continual Test-Time evaluation.\n\nThe authors need to make more comprehensive evaluations on the influence of the different components of their loss.\n\nLIMITATIONS:\nThe limitations section in the appendix is very short. A clear limitation is that the method introduces several hyperparameters which need tuning. This imposes an additional computational overhead which has not been discussed.\n\nAnother limitation which can be fixed is that the authors did not study the continual learning setting to analyze whether their method collapses when adapting for long time periods.\n\nIn 2024, most people work with large-scale pretrained models such as e.g. CLIP. It is not clear how this method could be used on those models and whether improving ResNet50's performance is relevant nowadays.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents REMA which designed to improve the robustness of deep learning models against out-of-distribution (OOD) data. REMA employs a selective slot-based reconstruction module to dynamically map dense pixels into a sparse set of slot vectors, enabling the identification of major components from objects in an unsupervised manner. Additionally, a hypergraph-based relational reasoning module is introduced to model high-order dependencies among these components, ensuring topological homogeneity. Experiments conducted on standard benchmarks demonstrate that REMA outperforms state-of-the-art methods in OOD generalization and test-time adaptation settings, highlighting its effectiveness in handling distribution shifts and enhancing the adaptability of deep models in real-world, non-stationary environments.\n\nSTRENGTHS:\n1. REMA introduces a unique combination of selective slot-based reconstruction and hypergraph-based relational reasoning to address OOD robustness, which has not been explored extensively in previous studies.\n\n2. The framework effectively identifies and leverages major components of objects without requiring human prior knowledge or fine-grained annotations, reducing the need for extensive labeled data.\n\n3. The paper provides extensive experimental results on multiple benchmark datasets, showing improvements over existing state-of-the-art methods in both OOD generalization and test-time adaptation scenarios.\n\nWEAKNESSES:\n1. The paper does not provide a detailed analysis of the computational overhead introduced by the new modules, which could impact the scalability of the approach for large-scale or real-time applications.\n\n2. While the experiments cover several benchmark datasets, the generalizability of REMA to other types of datasets or more diverse real-world scenarios is not fully explored.\n\n3. The effectiveness of REMA relies on several hyperparameters (e.g., the number of slots, attention iterations), and the paper does not thoroughly investigate the sensitivity of the model's performance to these parameters.\n\nQUESTIONS:\n1. Parameter Sensitivity: How sensitive is REMA's performance to the choice of hyperparameters such as the number of slots and attention iterations? Could an automated hyperparameter tuning method improve the robustness and generalization of the model further?\n\n2. Computational Efficiency: What is the computational overhead associated with the selective slot-based reconstruction and hypergraph-based relational reasoning modules? How does this overhead impact the scalability and real-time applicability of REMA in large-scale or resource-constrained environments?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose an approach to tackle OOD generalization via a method that tightly combines learning-based feature extraction with graph-based relationship modelling to explicitly learn and represent the topological structure of image data, achieving competitive results across a range of OOD and test-time adaptation benchmarks.\n\nSTRENGTHS:\n**Originality & Significance:**  \n- Interesting method focusing on explicitly modelling the often neglected /only implicitly modelled topological structure of image data via a mixture of learning- and graph-based methods\n- Proposed approach demonstrates an appealing way of ‘casting’ the common human intuition regarding our visual reasoning process, which draws on different levels of hierarchy, into an end-to-end trainable algorithm through a well-justified composition of different components\n\n**Quality:**  \n- Experiments conducted with a good selection of comparative methods to gauge performance improvements across two different tasks (OOD and test-time adaptation) and multiple datasets\n- The authors perform an appropriate ablation of their main components, both quantitatively and qualitatively \n\n**Clarity:**  \n- The paper is well written and easy to read and follow; The topic is well motivated and well-placed within the context of related efforts, clearly pointing out which areas of research the authors build upon and what the remaining challenge is;\n- Clear visualizations help the reader to quickly grasp the underlying concepts, both methodologically and architecturally\n\nWEAKNESSES:\n_TLDR; I do not see any severe ‘prohibitive’ weaknesses in this work, but have a few questions and requests that I’d like the authors to clarify & address._\n\n- Some insight into what the slots actually represent would be insightful and elevate the paper, see below.\n- Missing detail on complexity, please see question section below.\n- Missing details regarding inference and training procedure, see below.\n- Some inconsistencies in notation as well as some typos that should be corrected\n\nQUESTIONS:\n**Main concerns, questions & potential improvements:**\n\n**[Q1]**: Consistency in notation could be improved.   \n The authors initially use lower-case characters to indicate functions, e.g. in equation (1), and mathcal font to indicate sets (e.g. input, output, feature space/domain);   \nHowever, this suddenly changes during the introduction of the method (l.94),  and mathcal is used to refer to transformations (e.g. K_beta, which is nothing else than a function);   \nThen slightly later (l.135), mathcal refers to the ‘result’ of the linear transformation that essentially forms the graph nodes;   \n$\\textrightarrow$ I’d highly recommend to keep a consistent notation to avoid confusion (as it did confuse me); \n\n**[Q2]**: Some missing details regarding training & inference procedure. \n- How exactly is training performed? The authors mention that the “algorithm first trains deep models using the reconstruction objective” (l.171) – is the model (or a part of it) then frozen when the graph is created/HORR trained, or is HORR simply added and everything then trained/finetuned? The appendix section doesn’t make this clear to me either, and some more details here would be helpful.\n- The authors mention in l.99 that the queries “will be refined during the T attention iterations”. How many iterations are employed in practice, and is this performed for each new image pair (i.e. each step) during training? \n- How exactly is the inference actually performed at test time? Does inference require pairs as employed during training, or what is the exact setup there? \n\n**[Q3]**: Details around complexity;   \nFollowing up from Q2, what is the ‘complexity’ of the method in contrast to other related methods, e.g. in terms of inference time?  \nNote that given the impressive performance and graph-based nature, this could be an interesting insight to the reader (to gauge potential trade-offs between multi- and 1-step reasoning methods)  \n\n**[Q4]**: Details regarding slots / information bottleneck:   \n- The authors mention in the appendix that ‘5’ slots have been typically used. Is this the same for all datasets? And how many were selected as relevant?\n- Do these numbers change across datasets or even classes? And if so, could you provide possible insights into why, e.g. multiple objects / increased complexity / more components, or similar.\n\n**[Q5]**: Insights into the actual 'slot correspondence'/representation:  \n- 5 slots seems quite few to represent the content of an image. Is this due to the simplicity of the images, and do the authors have some intuition how this would change for `real-world' natural settings? \n- How do the slots actually 'align' with 'components' of objects: Taking the motivational picture of the horse, would your algorithm actually identify 'parts' of an object as a component, or rather different objects in an image, or entirely different? Some insight into these aspects would be highly interesting to actually see to which extent the inner workings align with what we would expect from humans (which is, after all, your underlying motivation) \n\n---\n\n**Additional comments:**   \n- typo l.92: embedding -> embeddings \n- capitalization l.109: we -> We\n- typo l.216: methods […] is -> are\n\nLIMITATIONS:\nLimitations have been adequately addressed by the authors in the appendix;   \n$\\textrightarrow$ I highly appreciate the authors being honest and providing ‘proper’ limitations to their method in terms of potential applicability!", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new methodology, REconstruct and MAtch (REMA) is introduced to learn a more robust and generalizable feature set in computer vision models.  REMA relies on a slot attention module to learn sparse embeddings of features which characterize a target object, and this module is then coupled with a high-order relational reasoning (HORR) module which creates a graph representation of the object.  This graph representation encodes how the sparse features relate to each other.  Because the feature encodings are both simplified through the sparse attention module and contain a learned topological relation from the HORR module, they are expected to be robust to domain drift. The robustness to domain drift is demonstrated using several benchmark datasets with consistent networks and training hyperparameters shared between them.  REMA consistently and clearly outperforms SOTA methods for OOD generalization; the results are convincing.\n\nSTRENGTHS:\nThe methodology is novel; the combination of the two modules represents a new contribution to the area of identifying robust features for computer vision.  The results reported are extensive, including latent space analysis of how features are represented, ablation studies, and comparison against $\\approx$20 other OOD generalization methods, with the proposed REMA method outperforming all of them.  An improved method for robust OOD model training is a significant contribution, and the work put into reporting the technical aspects of the methodology will encourage dedicated researchers to continue developing this approach to robust feature identification.\n\nWEAKNESSES:\nThe report of the proposed method is exceedingly technical.  Discussion of the intuition motivating the method is restricted to a brief statement concerning the human visual system, and no connection back to the human visual system is made throughout the remaining discussion of the methodology.  No further motivation or context is provided for algorithmic design choices, which prevents the reader from understanding *why* the method works well. The emphasis is entirely on the *how*. I suspect that a significant literature review was performed as part of the algorithm design; sharing the literature review context for these design choices would have helped motivate and clarify the methodology.  It could easily have been included in the appendix without detracting attention from the main contribution of the work. Alternately, the context could have been included in the main result, and the implementation details left to the appendix.\n\nThe benefit of the HORR module is open to some question.  The impact of the HORR module vs the SSR module is discussed in Section 4.3, Table 4, and shown in Figure 6 with the tSNE embeddings.  The results of Table 4 are reported without uncertainty, and the values are close enough (within 2%-8.5% of the baseline without REMA) to make uncertainty a valuable indicator of the role of each module. Given the stochastic variation inherent to the tSNE algorithm, Figure 6c (SSR w/o HORR) and 6d (SSR w/ HORR) could be considered identical since there are no repeated embeddings reported, and no figure of merit describing the variance for repeated embeddings and clustering is included.  Visually, the embeddings are very close, so although the value of the SSR module is clearly demonstrated in disentangling latent features in Fig. 6c, the value of the HORR module has some doubt.  (An example of suitable metrics for quantifying the author's claim of better clustering in Figure 6d vs. Figure 6c might be DBSCAN or OPTICS applied to multiple embeddings, and quantified with the mean and std of the v-measure score for the clusters.)  The supplemental Figures 6-8 do little to clarify the situation; the authors state only that SSR and HORR behave differently depending on the type of corruption. An additional place to clarify the behavior of the individual SSR and HORR modules would have been in Figure 4a, where the grad cam results with and without the SSR and HORR modules could have been reported.  Finally, the comment in the supplement regarding training the SSR module first suggests that although including the HORR module may be beneficial, it is not necessarily key to the success of the work as no mention of training HORR first is made.\n\nOverall, the results are extensive, and support the conclusion that REMA is an improvement on the SOTA for OOD robustness.  But the lack of contextualization and motivation for the algorithm in general, and the behavior of the different modules specifically, is a weakness of the paper.\n\nQUESTIONS:\n1. The stated goal of the Selective Slot-based Reconstruction (SSR) module is to create a sparse embedding of target features.  In section 3.3, it would seem that a variational auto encoder type approach is used to train the encoder.  Sparsity can already be considered forced due to the size of the feature vector/latent space of the VAE; a more sparse feature vector is just one that is smaller in this context.  How does the additional SSR module change the latent representations to make them more sparse?  Is it just removing specific frequency components?  Why is it not possible to do this directly on the latent space of the encoder through a specialized loss function without the use of an additional module?  \n\n2. In section 3.1 \"(A) standard MLP skip connection is applied\" after the GRU; why? What is the intuition here?\n\n3. What is the minimum and maximum connectivity of the hypergraphs constructed in Section 3.2?  How would this dimensionality relate to the dimensionality of the sparse embeddings?  Of the original feature vector extracted from the encoding module?\n\n4. Please provide citation(s) for the statement in Supplemental section C.1 \"sparse modeling based on slots may struggle to accurately separate the scene into several main components\".  \n\n5. In the supplemental section B.2, why is it reasonable to expect that the SSR and HORR modules have different effects based on the type of data corruption?  Shouldn't the keypoints of a target object (and therefore the sparse embedding of those key points as well as the topology of how those keypoints relate to each other) be unaffected by whether it is fog or frost corrupting the image?\n\nLIMITATIONS:\nThe authors report limitations in the supplemental material.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}