Neural Collapse Inspired Feature Alignment for Out-of-Distribution Generalization
Authors: Zhikang Chen
Min Zhang
Sen Cui
Haoxuan Li
Gang Niu
Mingming Gong
Changshui Zhang
Kun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Spurious Correlation
Neural Collapse
Out of Distribution
Primary area: other

Abstract: The spurious correlation between the background features of the image and its label arises due to that the samples labeled with the same class in the training set often co-occurs with a specific background, which will cause the encoder to extract non-semantic features for classification, resulting in poor out-of-distribution generalization performance. Although many studies have been proposed to address this challenge, the semantic and spurious features are still difficult to accurately decouple from the original image and fail to achieve high performance with deep learning models. This paper proposes a novel perspective inspired by neural collapse to solve the spurious correlation problem through the alternate execution of environment partitioning and learning semantic masks. Specifically, we propose to assign an environment to each sample by learning a local model for each environment and using maximum likelihood probability. At the same time, we require that the learned semantic mask neurally collapses to the same simplex equiangular tight frame (ETF) in each environment after being applied to the original input. We conduct extensive experiments on four datasets, and the results demonstrate that our method significantly improves out-of-distribution performance.

Neural Collapse Inspired Feature Alignment for Out-of-Distribution Generalization
Zhikang Chen' Min Zhang² Sen Cui Haoxuan Li
Gang Niu
Mingming Gong Changshui Zhang Kun Zhang
Tsinghua University East China Normal University 3 *Peking University *RIKEN
5 Institute for Artificial Intelligence, Tsinghua University (THUAI)
Beijing National Research Center for Information Science and Technology (BNRist)
Department of Automation, Tsinghua University
6 The University of Melbourne Carnegie Mellon University
Mohamed bin Zayed University Artificial Intelligence (MBZUAI)
Abstract
The spurious correlation between the background features of the image and its label arises due to that the samples labeled with the same class in the training set often co-occurs with a specific background, which will cause the encoder to extract non-semantic features for classification, resulting in poor -istiiiiiitiiittioonn generalization performance. Although many studies have been proposed to address this challenge, the semantic and spurious features are still difficult accurately decouple from the original image and fail to achieve high performance with deep learning models. This paper proposes a novel perspective inspired by neural col- lapse to solve the spurious correlation problem through the alternate execution of environment partitioning and learning semantic masks. Specifically, we propose to assign an environment to each sample by learning a local model for each envi- ronment and using maximum likelihood probability. that the learned semantic mask neurally collapses to the same simplex equiangular tight frame (ETF) in each environment after being applied to the original input. We conduct extensive experiments on four datasets, and the results demonstrate that our method significantly improves -atto--distribution performance.
the same time, we require
Introduction
The out-of-distribution (OOD) problem refers the fact that the training dataset and the test dataset have different distributions in different environments, after the training process is completed, what we want have correlation between semantic features and label, however, the presence of spurious feature(enviromment
information) can make false correlation between environmental information and labels. The semantic feature in an image refers to the object of the class, while the spurious feature in an image refers to the background or the environment. As shown in Figure [] the background color for digits 0 and predominantly orange, while the background color for digit 1 is predominantly green. In this figure, the semantic feature includes digits 0, 1 and 2, whereas the spurious feature includes orange and green backgrounds.
Recently, many OOD methods have been proposed to learn invariant representations for different
environments by introducing various regularization methods [Ajakan al., 2014 Arjovsky et al.
2019
Ahmed et al. 2020 Ahuja et al. 2021 Krueger et al. 202 Liu et al. 202 Tong et al.
2023
Zhu al. 2023a Although these methods achieve good OOD performance on data with
*Corresponding author. Email: hxli@ stu, stu.pku.edu.cn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

environment variations from the training dataset, on the one hand, they require a well well-paatitioned environment advance, but real datasets usually do not have environment labels; on the other hand, under cross-environment conditions, these methods do not align feature prototypes well between the test environments and the training environments. this paper, we equate semantic features with invariant features and spurious features with variable features.
We first explored the limitations of previous OOD learning methods from a new perspective and conducted toy example experiment, as shown in Figure The digits 0 and 2 are associated with orange background, and the digit is associated with green background. The phenomenon of neural collapse, which means that after sufficient training, the categories will collapse to a simplex ETF such that the angles between the feature prototypes of two neighboring categories are equidistant. Thus, in Figure !l after the training is completed. the feature prototypes of the digits 0, 1 and 2 occur three equal parts. Here, to measure the extent of neural collapse, we propose to compute the Frohenius norm (F-norm) of the difference between the feature prototypes and the standard simplex ETF. smaller F-norm indicates closer proximity to the standard simplex ETE. The ERM approach (Vapnik al., 1998] that leads to the failure of OOD generalization is due the endogenous nature of the class features not being able to align the class prototypes after training is complete different environments. Although IRM-based methods (Arjovsky et al., 2019] theoretically learn feature representation such that the last layer of the feature extractor is similar across environments, we empirically found that IRM-based methods are not aligned very well, which motivates using neural collapse to align feature for OOD generation.
In this paper, in order to bridge this gap, we believe that a feature extractor for OOD generalization across environments should ensure that semantic features are aligned to the same simplex ETF, and we have verified from our experiments that better alignment can significantly improve OOD performance. Specifically, our method can be applied both with and without environment labels. In the absence of environment labels, we can automatically partition the environments and assign local models to different environments. By predicting the variable components the input, we take the logits corresponding to the label (one-hot encoding) from the predictions of the local models in different environments, form a vector, and select the maximum value of the vector. The corresponding enviromment of this maximum value is assigned as the environment for the input. When the new environment shows minimal changes compared to the old environment, we consider the environment partitioning be complete. In Figure partitioning method. When environment labels are available, we learn masks to extract semantic components, firstly fix a simplex ETF classifier, and in different environments, for semantic features, we pull them all to the corresponding position of the same class prototypes, thus realizing the alignment operation.
provide detailed example illustrate environment
The main contributions of this paper are summarized as follows:
We explore the OOD problem from new perspective, namely the use of neural collapse to guide feature alignment for OOD generalization.
We explore the separation of semantic features and spurious features under conditions without environment labels, as well as with given environment labels.
We conduct extensive experiments on four publicly available datasets to validate the effec- tiveness of the proposed methods.
2
Related Work
ont-of-Distribution Generalization. Outtof-diitribbution (OOD) generalization has been a topic of significant interest in the field of machine learning and computer vision. Researchers have explored various approaches and techniques to improve the robustness and generalization capabilities of
models on unseen target domains. These OOD methods use different strategies to update the model,
mainly including metalearing (Li et al.) 2018a Zhang et al., 2023). domain alignment AAaakan al. 2014 Li et al., 2018b), regularized training [Zhang et al. 2021 Krueger et al. 2021 Shi et al.
2021 Xu and Jaakkola [202]], causal learning [Ahuja et al. Krueger et al. 2021 | Koyama and
Yamaguchi 2020), etc. For example, IRM (Arjovsky et al. [009], representative OOD method,
learns invariant representations with classifier optimal to domain changes as a regularization term.
SANDMask (Shahtalebi et al. 202T regularizes model training by updating the parameters in the
direction where the gradient components have consistent signs across domains. Although these

---

Training complete
Test process
457
4.31
Previous methods
(a)
2.0
3.50
1.16
Tanch
E101
IRM
Semantic Feature
+
Owss
OOI Mrrrri (N)
Spurious
96.6
92.8
Training complete prototypes
Eavl prototypes
Eas2 prototypes
5
OdverdMNINT
Figure I: Left Figure: A comparison between our approach and previous methods(conellde ERM and IRM-based methods). (a): The comparison of method, the ERM method, and the IRM method based on the F-norm metric. (b): The comparison of our method, the ERM method, and the IRM method in terms of OOD accuracy (%)
OOD methods have theoretically and experimentally demonstrated their effectiveness in learning domain-invarian features, most of them require the use of environment labels, which does not meet the requirements of real-world applications. Furthermore, we empirically found that the information extracted by the last layer of the feature extractor based on the IRM-based method in different environments cannot be well-aligned. In this paper, taking a step forward, we propose novel method that aligns the semantic features extracted by the OOD feature extractor to the same simplex ETF under neural collapse. Note that our method can be applied both with and without environment labels.
Neural Collapse The phenomenon known as neural collapse, first identified by [Papyan et al. 2020), refers to the observation that when the number of training samples across different classes is balanced, both the feature vectors in the final layer and the classifier vectors tend to converge to the
simplex ETF upon the completion of training. Recently, several studies [Ti al. 202T 7huu al.
2021 Tirer and Bruna
2022 Zhu et al., 2023b Li et al. 2023 Xie et al. 2023 Yang et al. 2023 2024 Sūkeník et al. 2024 have utilized this phenomenon to guide the training process in imbalanced data sets. Among
Beaglehole et al. 2024 Fisher et al. 2024 Guo al. [0224] Kothapalli
them. Yang et al. (2023) involves pre-allocating a fixed number of classes simplex ETF for continual learning, guiding the learning of minority classes in subsequent incremental steps, thereby ensuring convergence of intra-class features specified positions and maximizing and uniformly separating inter-class features. Xie et al. 2023 addresses class imbalance by designing novel
loss function, namely the attraction-rejection balanced loss. [i et al. 2023 apples neural collapse into federated learning scenarios. Under distributed conditions, neural collapse is used to guide the alignment direction of each client, and the personality each client model is maintained through fine-tuning. The aforementioned methods only consider collapsing the same class onto a single point in fixed simplex ETF without accounting for the impact of intra-class spurious correlations. We are the first to utilize the concept of neural collapse to address spurious correlations within classes, (OOD) performance.
thereby enhancing
3 Preliminaries
3.1
Out-of-Distribution Generalization
In the given dataset D == Y) = {(x;: typically, during the training process, the data set divided into a training set * and test set D where E is sampled from the training distribution (X, Y), and 1e is sampled from the test distribution Pte (X, Y). There is model that can be divided into feature extractor f((:wy) and classifier g(::w)). For -usti--distributioon (OOD) scenarios, the training distribution is not observable, and the training distribution differs from the test distribution, i.e., Per (X, Y) # (X, Y). Specifically, according to previous work, we have multiple environments in the training set, that is, E {ei, e2, ..., eg). ). In different environments, for
3

---

No Condition
Old Enw-New Env
Yes
Inpat(x, y, and)
Splitting Environments
New Env
Neural C Collapse for Learning Mask
y
O 2 Eav1
1
Select Env
X2
2.
Local Model
Env2
Fixed Classifier
Figure 2: The overall framework our method. Left Figure: Input information, including figure, label and environment. Middle Figure: In the scenario of unknown environments, the process of partitioning the environment. Right Figure: Utilizing neural collapse to guide learning of masks for extracting invariant components.
X in the training set, we can divide it into semantic parts and spurious parts, the correlation between these two unstable, thus it prone to generating spurious correlations.
3.2 Neural Collapse
Neural collapse refers to phenomenon observed during the final stages of training balanced data (post-zero training error) [Papyan et al.) 2020]. It reveals simplex ETF structure formed by the last-layer features and the classifier, which can be defined as:
Definition (Simplex Equiangular Tight Frame). simplex Equiangular Tight Frame (ETF) refers to matrix that is composed of K vectors ER", d 2 K and satisfies:
K U
1 IK
=
V K-1
K
where = =Vvi, V2..., VK) E RexK, U RaxK allows a rotation and satisfies U = Ix. IK represents identity matrix, and l stands for all one matrix.
For any V E Vii [1, K] and satisfies:
K
V.,; =
E E K1,
(2)
K
K-
K-1
where 81.3 equals to 1 when i j. otherwise is equal 0. for all v, V,i [1, there the same L2 norm, and the inner product between any two vectors
Neural Collapse. The phenomenon of neural collapse is guaranteed by the following four findings: variability collapse (NC1), convergence the simplex equiangular tight frame (NC2), convergence to self-duality (NC3), and simplification to the nearest class center (NC4).
NC1: During the final stages of model training, the final layer feature vectors of the same class will collapse to the class mean, i.e., for all k. Σt - 0. S The D (fr, fx) (fr. fe) where ng represents the number of samples in class k, fe. denotes the feature obtained from the i-th sample of class k, fü represents the mean feature of class k.
NC2: The class means of all classes will converge the vertices of simplex ETF centered on the global mean, defined in Definition [ î (f, fa) [f. fall, where fo Σ NN fr.i represents the global mean of the feature.
NC3: The feature prototypes centered on the global mean will align with the corresponding classifier weights, which means that the classifier weights converge to the same simplex ETF, i.e. f. Va! [|v where Va represents the classifier weight for class k.
Through the above points, it can be ensured that the network classifier converges to the nearest class center, i.e, arg max; (f, Vε) arg ming ||f fell, where f represents the features of the sample.
NC4:

---

4 Methodology
In this section, we delve into comprehensive discussion of our proposed method, NCFAL, along with its specific implementation. Section [4.1 outllnnes the framework of the proposed method for out-of- offooiiittibuttion based on neural collapse. We use fixed ETF classifier to guide the alignment of invariant features across different environments, facilitating better learning of masks to separate invariant and variable features. In Section 4.2 we provide detailed explanation of how environments are partitioned. In Section 4.3] we elaborate on how a fixed ETF classifier is used to guide mask learning. In Section 4.4 after obtaining invariant features using the trained mask, we input them into the neural network for subsequent training until convergence.
4.1 The framework of proposed method
Neural collapse reveals the optimal geometric structure of the classifier and feature prototypes after sufficient training (i.e., the simplex ETF). It inspires us to use a simplex ETF as a fixed classifier from the beginning, guiding the alignment of invariant components across different environments. Therefore, we propose a novel algorithm for the OOD generation inspired by neural collapse, NCFAL Specifically, described in Section 4.2 in real-world scenarios, environment information is often unknown. Hence, we need to automatically partition environments, and when certain conditions are satisfied, we consider the environment to be sufficiently to improve model generalization, after obtaining environment information. we guide the alignment of invariant components across environments using a fixed ETF classifier. Given that class imbalance is likely across different environments, we employ a loss function to mitigate the impact of this imbalance. We alternate between Section 4.2 and Section 4.3 enabling the model to learn improved masks for partitioning invariant components. In Section 4.4 after learning the mask, we obtain semantic features for subsequent training of the predictive model.
partlitaationd As described in Section (4.3
Specifically, as shown in Figure 2 our methodological framework is illustrated. Initially, when x is input, it passes through 1 for the separation of variable components, allowing for prediction using corresponding models in different environments. Subsequently. an environment selection process assigns the input designated environment, yielding new environment label. When the difference between the new environment label and the previous one is less than a given threshold or specified number of partitioning iterations reached, the mask learning process begins. In this phase, x from different environments is input. Since invariant features are extracted, they should collapse onto the same simplex ETF in any environment. This approach guides the alignment of invariant features across variable environments.
4.2 The implementation of environment partitioning
This chapter focuses on environment partitioning in scenarios where environment labels are unknown. The partitioning process relies on the intermediate module depicted in Figure which takes the input information and outputs a new set of environments E, where each environment reflects a type of spurious correlation present in the input. We divide the input into semantic and spurious features, using the invariant mask m to distinguish between them, resulting in semantic features +(x) and C (x) (1 m) @ Subsequently, the variable features V (x) are input into local models, which are associated with the environments, for identification. We designed two-stage partitioning method to iterative partition environments until convergence.
spurious features L (x): (xx)
Environment Local Model Learning Stage: Let X!" be the interaction set environment e. We aim to use local model to represent the environment e. Intuitively, the primary distinction between different environments lies in their interpretation of spurious correlations. Therefore, we model environments based these spurious correlations. Specifically, for the interactions in environment we predict using variable representation learning model r(c), In other words, to describe environment e, we learn the predictive model as follows:
arg min ((r(t) v (w"))
where t represents the training data corresponding to environment e, we represents model parameters corresponding to environment e.
5

---

Condition
No
Yes
Eav=Neww Env
Logits
Argmax
New Emv
Projection Layer
Y1
0]
Feature Extractor
X2
y
0
1
1
1
Y2
E1
E2
10.2
10.4
10
0.2
0.11
01
[0.5
E1 E2
O 2 Enyl
m
O
X
2
10.1
0.88
10.6
E1
E1
(0.2 0.1 9.7]
10.7
Eav2
Local Model
0.2
Fixed EIF Classifier
Figure 3: A comprehensive elucidation of Figure is provided herein. Middle Figure: An illustrative example presented demonstrate the process of environment partitioning. Right Figure: An in-depth exposition of the process of learning invariant masks is provided.
Environment Partitioning Stage: In this stage, we have already developed environment models that be used assess the spurious correlations in different environments. Consequently, interactions should be separated based on these spurious correlations. To distinguish environments and the interactions within the input, we employ the following formula, which determines that interaction belongs to the environment with the highest probability of identifying it. Note that predictions are based on the variable component (x), which represents spurious information. The maximum value of this tensor is selected, indicating that the variable features of the input are more distinguishable in the environment corresponding the maximum value. Thus, the input assigned to that environment.
e(x) = arg max ((e) V (We).
(4)
eEE
where for any environment E &, select the one with the maximum logit the corresponding environment for input x.
We illustrate this environment selection process with example in Figure Considering logit 0 with orange background, when is inputted into different environment models, the corresponding predictions are [0.5, 0.3, 0.2] in environment and [0.4, 0.5, 0.1] in environment 2. By extracting the logits at the positions corresponding to the labels(one hot encoding), we obtain vectors [0.5, 0.4). Taking the maximum value, we can determine the corresponding environment environment 1.
4.3 The implementation of learning masks
This chapter primarily addresses OOD generalization by learning masks to separate invariant repre- sentations when environment labels are known. Our approach is guided by the perspective of neural collapse to align invariant representations across environments to pre-fixed simplex ETF, As shown in Figure 3 once suitable environment partitioning mask is learned, we apply it to the input x to obtain the invariant components. Thus, in different environments, for the semantic features (xx) in all e,, Ej E supp(E), we have: pe; (Y (x)) pe; (Y (x)).
In different environments. we use corresponding models to align the invariant features. First, we fix standard simplex ETF as the alignment direction. Given the potential issue of the number of different classes imbalance across different environments, we use balanced loss assign weights proportional to the quantity of each class for the training process:
n2 exp(3 v(T
Lmaak llg
& K,EE,
Ee n. exp(8 v.f)
f f=
(h,,,,),h = (000==f(((x))) (()()),)))
where Th..k represents the number of samples of class l: in environment Va E V represents the fea- ture vector corresponding to class k on the fixed simplex ETF V. represents the normalized feature v obtained after projection mapping the original feature h, B represents the learnable temperature, represents the sample balaneing parameter, W/ represents the parameters of the feature extraction module in the model, wp represents the parameters of the mapping layer in the model. When learning the mask, add some random noise initialize the mask:
m, max min(1, m, e)}, e N(0, m, E m.

---

After completing the above training steps 5 we perform a clipping operation on the mask to fix the range m,:
mi max (0, min (1, m, + ++)] m, m.
4.4 The implementation of learning predictive model
In this chapter, we apply the environment partitioning and mask learning in Sections 4.2 and 4.3 respectively to the input data to extract semantic features. These features are then fed into the predictive neural network for training until the network converges:
arg min C(T (x, +lw")[x*
where w" represents the parameters of the final predictive model.
Summarily, the overall training process is presented in Algorithm [ The steps in the Splitting environments process correspond to Section (.2] while the Learning mask process corresponds to Section After convergence, semantic features are used to train the predictive model, corresponding to Section
thereby achieving better OOD generalization.
Algorithm The overall training process.
Data: X for splitting environments, learning mask and training process. Result: Predictive Model |" (x w", () for the final predictive process. for i 4 1 7 do
Splitting environments if & is unknown then do
for do Optimize ((e) via the local model training process X in Eq. 3); end
for E do
Compute x via the highest probability of identification as Eq. end while Converged;
end
Learning mask do
Learn m via aligning feature prototypes the fixed simplex ETF in Eq. (5):
while Converged;
end
1 Training process Optimize 1 (x w", 2) with semantic components ((x) via Eq. (6);
#1
Experiments
5.1 Experiment Setup
Datasets. Following the work (Gulrajani and Lopez-Paz, 2020 we evaluate our method with baselines benchmark datasets, using four datasets, namely ColoredMNIST, ColoredCOCO. CO- COPlaces and NICO. ColoredMNIST is colorized on the MNIST dataset. The image dimensions were set [2, 28, 28], the digits [0, 2, 3, are designated as category while the digits [5, 7, 8, 9] are designated as category 1. ColoredCOCO dataset is derived from the COCO dataset, which includes selection of categories. Background color alterations were applied using different colors. All images are configured with dimensions of (3, 64, 64), COCOPlaces employs the same classes and settings as ColoredCOCO, with the distinction that we sample images from Places as spurious information. NICO dataset is a real-world dataset, including 10 subclasses for animals and subclasses for vehicles. In total, our split consists of 4,080 samples of dimension (3, 224, 224) and 2 classes of the classification task. More details can be found in Appendix A. For each dataset, we partition it into two subsets, d, and d2, based on the environment, with the ratio of sample quantities between dj and d2 being 9. Subset dj from the training environment is used for

---

model training, while d2 is employed for testing the models within the training environment. In the testing environment, d1 is utilized assess the out-of-distribution (OOD) performance of the models, while d2 is utilized according to DomainBed standards for selecting the best model.
Architecture. On ColorMNIST, training is conducted using 4-layer convolutional neural network. For the ColoredCOCO and COcOPlaces datasets, we adhere to the setup outlined in Ahmed et al. 2222)) Gulrajani and Lopez- Paz [2020), employing ResNet8 for training. On the NICO dataset, training is performed using ResNet18.
Baselines. In order to demonstrate the advantages and effectiveness of approach, conduct
comparative tests against different OOD learning methods. including (1) IID learning: ERM [Vapnik]
et al.
1998 (2) OOD learning ( sixteen methods): IRM [Arjovsky et al. 2019]. VREx |Kueeer et al.
2021
ARM (Zhang et al.] [022]] GroupDRO (Sagawa et al., 2020). MLDG [L et al.
2018a MMD
[Li al.
2018b] IGA [Koyama and Yamaguchi 2020), SANDMask (Shahtalebi et al., 2021], Fish
(Shi et al.
2022].
CDANN [Li et al. [018c]. TRM [uu and Jaakkola
2021]. IB_ERM [Ahuja et al.)
2021
IB IRM [Ahuja al. 2021]. CondCAD |Ruan al. 2021]. CausIRL.CORAL [Chevalley
et al.
2022].
MAP (Zhang et al. 2023
5.2
The comparison of OOD accuracy (%) between our method and other approaches.
Table presents a comparison between our method, the ERM algorithm, and 16 other OOD algorithms. evident that our approach outperforms all methods under the given environmental conditions. Moreover, in situations where the environment is unknown, our method demonstrates adaptive capabilities in partitioning the environment, achieving superior OOD performance. The reason behind this phenomenon, we believe, lies that sometimes manually partitioning the environment may not be optimal for neural networks to identify and discern spurious correlations, however, Utilizing neural networks partition the environment based on predictions might be more adapt at separating semantic and spurious features. Particularly for COCOPlaces, due to the complexity of backgrounds, manual partitioning may not always achieve appropriate segmentation. Therefore, from the results, when the environment is known, the OOD accuracy is 33.3%; however, using our method for environment partitioning yields a result of 36.7%.
Table 1: Average accuracy (%) of OOD on three toy and one real datasets using different methods.
ColoredCOCO ColoredMNIST
COCOPlaces
NICO
ERM
Vapnik et al.
1998
51.5
0.9
20.1
73.6 1.9
IRM
Arjovsky al.
2019
60.3 2.8
49.2 0.3
27.1+ + 0.9
75.8 + 2.0
VREx
Krueger et
2021
52.9 1.2
0.7
2.7
76.9
GroupDRR
Sagawa ot al. 2020
38.5 1.5
4.6
2.69
74.6 2.4
MLDG Li el al. 2018a
50.6 2.6
1.89
1.56
2.7
MMD Li et al. 2018b
5.84
2..3
78.2 1.2
IGA Koyama and Yamaguchi
2020
0.1
1.
0.3
1.3
SANDMask Shahtalebi
2021
58.6 6.5
1.2
2..9
72.8
Fish (Shi et 2022
28.0
0.5
2.1
77.0
CDANN Li at. |2018c
44.2 3.5
1.5
1.
72.8 1.8
TRM Xu and Jaakkola 2021
4.6
1.1
73.0 0.9
IB ERM Ahuja el al. 2021
0.2
1.14
2..2
77.7 1..
CausIRL. CORAL Cheval ley et al. 2022
28.7 1.3
1.1
26.1
75.7 +: 0.9
CondCAD Ruan of all 2021
49.2 0.5
4.72
2.3
73.9 1.4
IB MRM
Afuja a 2021
53.8 1.8
3.6
14.
7
ARM Z
Chang
2021
28.1
3.6
2.2
7644 1.6
MAP Zhang et 2023
52.6 0.5
5.
26.9
76.8 1.4
Ours
66.4 0.2
58.0 0.2
33.3 + 1.7
85.4 0.6
Ours (w/o env)
66.9 2.4
1..9
36.7 0.9
85,9 0.2
5.3
Ablation Studies
The comparison between other methods employing masking techniques and our method. Table presents the effectiveness of our proposed method for learning masks, we conducted comparisons with the IRM and REx methods, which incorporate regularization to facilitate learning of the mask through the gradient or variance. In contrast, our approach leverages phenomenon of neural collapse to guide the optimization direction for semantic feature enhancement instead of regularization. Simultaneously, we compared our method with the ERM approach, confirming the superiority of using masks acquire the invariant parts.
8

---

Table 2: The OOD accuracy (%) of our method and other regularization methods three datasets.
ColoredMNIST
ColoredCOCO
COCOPlaces
Ours
66.4 = 0.2
58,0
3 1.7
Ours (w/o env)
66.9 + 2.4
56,9 + 1.1
36.5 + 0.8
REx
61.3 + 3.0
56.0 +.2
27.4 1.3
REx (w/o env)
57.9 + 2.0
52.9 1.1
28.2 1.7
IRM
64.4 + 0.2
51.8
32.2 + 0.8
IRM (w/o env)
65.2 + 0.4
52.8
30.7 + 1.7
ERM
51.5 = 0.1
45.4 0.9
20.1 + 0.7
The comparison between mask is applied at the pixel level and the feature level. To validate the generality of our method, we conducted comparative experiments at both the pixel level and the feature level in Table S It was observed that sometimes extracting invariant features at the feature level could yield better results. In particular, improved performance was observed on the ColoredCOCO and COCOPlaces datasets. This phenomenon primarily stems from the fact that feature-level information high-dimensional, capturing relationships that are difficult discern.
Table 3: The OOD accuracy (%) of our method based on pixel and feature level on three datasets. ColoredMNIST ColoredCOCO
COCOPlaces
Ours
66.4 + 0.2
58.0 + 0.2
33.3 + 1.7
Ours (w/o env)
66.9 2.4
56.9 1.1
36.5 0.8
Feature
58.7 + 2.8
63.9 + 0.5
43.7 + 0.7
Feature (w/o env)
54.0 0.3
62.7 1.8
44.1 + 0.1
The comparison between randomly and using maximum likelihood probability method to split environments. Compared with random environment splits in Figure4aa a), method, which involves assigning different models to distinct environments and selecting maximum likelihood probabilities outputted by the models, found effectively partition environments.
0.7
0.669
Ours
Random
0.7
0.667
0.6
0.622
0.614
0.616
00D Accuracy
OOD Accuracy
0.574
0.5
0.417
0.5
0.4
0.3
0.4
0.318
0.289
0.2
0.3
0.265
0.283
0.1
0.2
ColoredMNIST
COCOPlaces
ColoredMNIST
COCOPlaces
(a) The method for selecting environments
(b) The number of environments to partition
Figure 4: Comparative experiments for randomly and the different number divided environments.
The comparison between different number of environment divisions. Compared with different number environment splits in Figure 4b). we can find that when the number of divided environ- ments is closer the actual number of given environments, better performance is achieved, thereby validating the effectiveness our environment division method.
Conclusion
We explore a new perspective understand the inherent limitations ERM and IRM-based methods, which fail due to the inability to align semantic features across environments, resulting in reduced generalization performance. By leveraging the phenomenon of neural collapse to guide the alignment of semantic features across environments. Compared to other OOD methods, we have significantly improved OOD performance. Moreover, in real-world scenarios where environment labels are unknown, our method addresses this by training local models to different environments, automatically achieving environment partitioning. This method greatly reducing the cost of manual annotation and expanding the applicability of our method. Additionally, will investigate and improve more

---

effective and efficient environment partitioning techniques in future work. We can also explore applying this method in other domains, such as segmentation or few-shot spurious correlation issues.
Acknowledgement
This work was supported in part by the National Natural Science Foundation of China (No. 623B2002) and the National Natural Science Foundation of China (No. 62176132). MG was supported by ARC DE210101624, ARC DP240102088, and WIS-MBZUAI 142571. Sen Cui would like to acknowledge the financial support received from Shuimu Tsinghua scholar program.
References
Faruk Ahmed, Yoshua Bengio, Harm Van Seijen, and Aaron Courville. Systematic generalisation with group invariant predictions. In International Conference on Learning Representations, 2020.
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Jean- Christophe Gagnon-Audet, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of- distribution generalization. Advances in Neural Information Processing Systems, 34:3438-3450, 2021.
Hana Ajakan, Paseal Germain, Hugo Larochelle, François Laviolette, and Mario Marchand. Domain- adversarial neural networks. arXiv preprint 101iv::412.4446, 2014.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv: /907 0009900999,,... 2019.
Daniel Beaglehole, Peter Súkeník, Marco Mondelli, and Mikhail Belkin. Average gradient outer product as mechanism for deep neural collapse. arXiv preprint arXiv:2402. 13728, 2024.
Mathieu Chevalley, Charlotte Bunne, Andreas Krause, and Stefan Bauer. Invariant causal mechanisms through distribution matching. arXiv preprint arXiv:2206. /646, 2022.
Quinn Fisher, Haoming Meng. and Vardan Papyan. Pushing boundaries: Mixup's influence neural collapse. arXiv preprint arXiv:240..06171, 2024.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Li Guo, Keith Ross, Zifan Zhao, Andriopoulos George, Shuyang Ling, Yufeng Xu, and Zixuan Dong. Cross entropy versus label smoothing: A neural collapse perspective. arXiv preprint arXiv:2402.03979. 2024.
Wenlong Ji, Yiping Lu, Yiliang Zhang, Zhun Deng, and Weijie Su. An unconstrained layer-peeled perspective on neural collapse. arXiv preprint arXiv:2110.02796, 2021.
Vignesh Kothapalli, Tom Tirer, and Joan Bruna. A neural collapse perspective on feature evolution in graph neural networks. Advances in Neural Information Processing Systems, 36, 2024.
Masanori Koyama and Shoichiro Yamaguchi. When is invariance useful in out-of-distribution generalization problem? arXiv preprint arXiv:2008. 01883, 2020.
David Krueger, Ethan Caballero, Joerk-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. olation (rex). In International Conference on Machine Learning, pages 5815-5826. PMLR, 2021.
generalization via risk extrap-
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy Hospedales. Learning to generalize: Meta-learning for domain generalization. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018a
Haoliang Li, Sinno Jialin Pan, Shiqi Wang, and Alex C Kot. Domain generalization with adversarial feature learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5400-5409, 2018b.
10

---

Ya Li, Xinmei Tian, Mingming Gong, Yajing Liu, Tongliang Liu, Kun Zhang, and Dacheng Tao. Deep domain generalization via conditional invariant adversarial networks. In Proceedings of the European conference on computer vision (ECCV), pages 624 639, 2018c.
Zexi Li, Xinyi Shang, Rui He. Tao Lin, and Chao Wu. No fear of classifier biases: Neural collapse inspired federated learning with synthetic and fixed classifier. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5319-5329, 2023.
Yong Lin, Hanze Dong, Hao Wang, and Tong Zhang. Bayesian invariant risk minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16021-16030, 2022.
Jiashuo Liu, Zheyuan Hu, Peng Cui, Bo Li, and Zheyan Shen. Heterogeneous risk minimization. In International Conference on Machine Learning, pages 6804-68 68-464144 14. PMLR, 2021.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Yangjun Ruan, Yann Dubois, and Chris Maddison. Optimal representations for covariate shift. arXiv preprint arXiv:2201.0000 2021.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why overparameteriiation exacerbates spurious correlations. In International Conference on Machine Learning, pages $346-8356, PMLR, 2020.
Soroosh Shahtalebi, Jean- Christhhst Gagnon-Audet, Touraj Laleh, Mojtaba Faramarzi, Kartik Ahuja, and Irina Rish. Sand-mask: An enhanced gradient masking strategy for the discovery of invariances in domain generalization. arXiv preprini arXiv:2 106.02266, 2021.
Yuge Shi, Jeffrey Seely, Philip HS Torr, N Siddharth, Awni Hannun, Nicolas Usunier, and Gabriel Synnaeve. Gradient matching for domain generalization. arXiv preprint arXiv:2104.09937, 2021.
Peter Súkeník, Marco Mondelli, and Christoph H Lampert. Deep neural collapse is provably optimal for the deep unconstrained features model. Advances in Neural Information Processing Systems, 36, 2024.
Tom Tirer and Joan Bruna. Extended unconstrained features model for exploring deep neural collapse. In International Conference on Machine Learning, pages 1148-211500 PMLR, 2022.
Yunze Tong, Junkun Yuan, Min Zhang, Didi Zhu, Keli Zhang, Fei Wu, and Kun Kuang. Quantitatively measuring and contrastively exploring heterogeneity for domain generalization. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 2189-2200, 2023.
Vladimir Naumovich Vapnik, Vlamimir Vapnik, al. Statistical learning theory. 1998.
Liang Xie, Yibo Yang, Deng Cai, and Xiaofei He. Neural collapse inspired attraction repplssion balanced loss for imbalanced learning. Neurocomputing, 527:60-70, 2023.
Yilun Xu and Tommi Jaakkola. Learning representations that support robust transfer of predictors. arXiv preprint arXiv:21 10.00940, 2021.
Yibo Yang, Haobo Yuan, Xiangtai Li, Zhouchen Lin, Philip Torr, and Dacheng Tao. Neural collapse inspired feature-classifier alignment for few-shot class incremental learning. arXiv preprint arXiv:2302.03004, 2023.
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt to domain shift. Advances in Neural Information Processing Systems, 34:23664-23678, 2021.
Min Zhang, Junkun Yuan, Yue He, Wenbin Li, Zhengyu Chen, and Kun Kuang. Map: Towards balanced generalization of iid and ood through model-agnostic adapters. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11921-11931, 2023.

---

Didi Zhu, Yinchuan Li. Yunfeng Shao, Jianye Hao. Fei Wu, Kun Kuang, Jun Xiao, and Chao Wu. Generalized universal domain adaptation with generative flow networks. In Proceedings of the 3t ACM International Conference on Multimedia, pages 8304-8315, 2023a.
Didi Zhu, Yinchuan Li, Min Zhang, Junkun Yuan, Jiashuo Liu, Kun Kuang, and Chao Wu. Bridging the gap: neural collapse inspired prompt tuning for generalization under class imbalance. arXiv preprint arXiv:2306. 130555230 2023b.
Zhihui Zhu, Tianyu Ding, Jinxin Zhou, Xiao Li, Chong You, Jeremias Sulam, and Qing Qu A geometric analysis of neural collapse with unconstrained features. Advances in Nearal Information Processing Systems, 34:29820-29834 2021.
12

---

A Implementation Details
A.1 Datasets
ColoredMNIST, introduced by IRM for assessing spurious correlations in out-of-distribution (OOD) problems, follows the configuration outlined in Zhang et al. 2023]. In accordance with this setup, we colorized the MNIST dataset with red and green hues to establish a strong correlation between digits and colors. The image dimensions were set to [2. 28, 28]. and the correlation coefficients for two training sets and one test set were specified as (0.9, 0.8, 0.1), indicating the proportion of red and green within category O. The digits [0, 1, 3, are designated as category 0, while the digits [5, 7, 8, 9] are designated as category 1.
ColoredCOCO dataset is derived from the COCO dataset, which includes selection of ten categories: airplane, bird, boat, bus, dog. horse, motorcycle, train, truck, and zebra. Background color alterations were applied using ten different colors. Their RGB values are [0. 100, 0]. [188, 143, 143], [255, 0, 0J. [255, 215, 0]. [0, 255, 0J. [65, 105, 225], [0, 225, 225], [0, 0. 2551, [255, 20, 147] and [160, 160. 160]. The number of samples for each training environment is 400 for each class but the testing environment is 200 for each class. All images are configured with dimensions of (3, 64, 64).
COCOPlaces employs the same classes and settings ColoredCOCO, with the distinction that sample images from Places as spurious information [Liu et al. 2021], such as b/beach, c/canyon, b/building facade, s/staircase, d/desert/sand, c/crevasse, b/bamboo forest, f/forest/broadleef, b/ball pit and o/oast house. Moreover, some random places are also used, i.e., k/kasbah, I/lighthouse, p/pagoda, r/rock arch, w/water tower, w/waterfall, z/zen garden.
NICO dataset is a real-world dataset including photos of animals and vehicles captured wide range of contexts (or backgrounds). There are 10 subclasses for animals and 9 subclasses for vehicles, with each subclass having 9 or 10 different contexts. Following (Lin et al. 2022], we select a subset of this dataset to simulate the spurious correlation of different contexts and classes (animal or vehicle), which is similar to the setting of ColoredMNIST. More specifically, we make use both classes that appear in four overlapped contexts: "on snow", "in forest", "on beach" and "on grass" to construct two training environments and one testing environment. In total, our split consists of 4,080 samples of dimension (3, 224, 224) and 2 classes of the classiffcation task.
2
7
5
8
5
8
4
2
7 3
9
9
6
(a) ColorddMISS
(b) ColoreCCOC
(c) COCOMaces
(c) NICO
Figure 5: Illustration of the ColoredMNIST, ColoredCOCO, COCOPlaces, and NICO datasets.
A.2 Impact Statements
We propose a new perspective for addressing the OOD generation problem by using neural collapse align semantie features across different environments. Our method can be applied both when environments are given and when they are unknown, as it can autonomously partition environments. This approach enables the model to independently learn semantic features, thereby achieving better generalization in future practical applications.
A.3 Devices
In experiments, conduct all methods on local Linux server that has two physical CPU chips (Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz) and 32 logical kernels. All methods are implemented using Pytorch framework and all models are trained on GeForce RTX 2080 Ti GPUs.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have done this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in conclusion part.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
14

---

Our work studies the algorithm for provides empirical evidences.
Justification:
Generation, and
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We have done this work in manuscript.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification:
We have strictly adhered to the anonymity guidelines when uploading the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ccm ec/
http://uuddee/oeeububieesseeee oolccyyooolyyyoooooooloooompilil for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/nips. //nips.cc/ppl http:ubblcgguuueeee
Stt.Subbiiiiioooiciic...o onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have provided the training and test details in main text.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have done this work in our paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided computer resources appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We are make sure preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed in boarder impact in appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This work focuses on addressing the OOD generation, which is not related to the misuse risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification:
We public datasets in our experiments.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the datasets..papersstt wwwppswipttttt.. thoode. com/ datasets package should be provided. For popular datasets, has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We don't publish new data assets in this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We conduce experiments on existing benckmark datasets.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work is not related to human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
19


Reviews:
[{"review_id": "UN06jVS7OE", "reviewer": "Reviewer_Tf9A", "summary": "The spurious correlation between image background features and their labels is a significant research problem, and the existing research suffers from the issue of difficult decoupling. In this paper, we propose a new approach to solve the spurious association problem by alternately performing environment segmentation and learning semantic masks from the perspective of neural collapse. Extensive experiments are conducted on four datasets and the results show that the proposed method significantly improves the out-of-distribution performance.", "questions": "For more information see Weaknesses.", "limitations": "Yes, the authors have adequately described the limitations in their submission.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper explores an important and widespread problem in real-world applications with solid and extensive experiments. The writing is clear and the narrative is easy to follow, facilitating an understanding of the spurious correlations problem. The use of neural collapse is particularly innovative.", "weaknesses": "W1:  In lines 48-50, it is mentioned that IRM-based methods learn similar representations from different environments, indicating a lack of proper alignment. Could you provide a corresponding experiment to demonstrate this phenomenon?\n\nW2: In Figure 3, the explanation of the middle module that uses logits to judge the environment is unclear. Could you please clarify the structure of the local models, the number of local models used, and the specific meaning of the logit values?\n\nW3: Could you explain the differences between masks based on pixel-level and feature-level approaches? If using feature-level masks, what is the impact of different network-layer features on model performance?\n\nThis work addresses an important and interesting question by introducing neural collapse from an invariant perspective, which I believe can provide valuable insights to the community. However, my main concern is that the same mask is used to learn both invariant and variable feature information. What are the advantages of the mask learning mechanism proposed in this paper compared to HRM's [1] mask mechanism?\n\n[1] Heterogeneous Risk Minimization", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "****Respond to W1:**** Thank you for pointing out the issue. We indeed omitted the comparisons in our manuscript. We have actually demonstrated this in Figure 1, where we used the F-norm to measure the degree of alignment. A smaller F-norm indicates that, after training, the feature prototypes are closer to the standard ETF.\n\n****Respond to W2:**** Thank you for the reviewer's reminder. We sincerely apologize for the misunderstanding caused by our negligence. A precise description will be provided in future revisions.\n\nAs shown in Figure 3, our method represents the variable features of the input as $(1-m) \\odot x$. By using $(1-m)$, we obtain the variable features (equivalent to the environmental information) from the previously partitioned data. The neural networks corresponding to different environments are then trained on these variable features. After training, the neural networks from different environments predict the respective logits. For each input’s true label, the index with the maximum value determines the environment of the input. The structure of the local model is the same as the local model used for alignment with the neural collapse algorithm later.\n\n****Respond to W3:**** Thanks for the valuable comments. We would like to explain as follows:\nThe primary difference between the two methods is the location of the masking. Pixel-level masking is applied to the input image, offering stronger interpretability. Feature-level masking, on the other hand, applies masking to the output features of the neural network, achieving better results and reducing the dimensionality of the masking. To address the reviewers' concerns, we performed experiments by applying masking to the output features of different layers in the network.\n\nMethods\t|ColoredCOCO|\tCOCOPlaces|\n| -------- | ------------------ | ----------------- |\nOurs\t|63.9±0.5\t|43.7±0.7\nOurs layer0\t|62.3±0.6\t|41.2±0.5\nOurs layer1\t|64.3±0.3\t|44.5±0.3\nOurs layer2\t|61.2±0.2\t|40.2±0.1\nOurs layer3\t|63.2±0.3\t|43.2±0.6\n\n****Respond to W4:**** Thanks for the valuable comments. We would like to explain as follows: Our method aligns invariant features across different environments by leveraging the phenomenon of neural collapse. For the same category, invariant features should be similar; thus, we use the same ETF for alignment, facilitating mask learning. Although the HRM algorithm also employs a two-stage approach, it partitions environments through clustering and then learns the mask through regularization. We conducted comparative experiments with the HRM algorithm to evaluate our method.\n\nMethods\t|ColoredMNIST | ColoredCOCO| COCOPlaces|\n| --- | ---|--- | --- |\nOurs\t|66.9±2.4|56.9±1.1\t|36.5±0.8\nHRM\t|61.3±0.6 | 52.7±0.7\t|33.4±0.6", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3NrhDu3L47", "reviewer": "Reviewer_F1YK", "summary": "The paper leverages the neural collapse inspired ETF behavior to simulate different environments in datasets, and uses it for OOD classification.", "questions": "The following papers seem missing from the neural collapse literature that may be helpful: \n\n1. https://arxiv.org/abs/2112.15121\n2. https://proceedings.neurips.cc/paper_files/paper/2023/hash/b63ad8c24354b0e5bcb7aea16490beab-Abstract-Conference.html\n3. https://openreview.net/pdf?id=162TqkUNPO", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The paper uses a phenomenon that's apparent in the standard setting, for a task that varies from the standard setting. It uses intuitive notions to tackle the task of OOD classification. The paper experiments are generally convincing.", "weaknesses": "The paper seems generally consistent and well merited. The experiments are a bit lacking, but are convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Respond to Q1:** Thank you for the reviewer's suggestions. We will include citations to these papers in a subsequent version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o9GoweowjQ", "reviewer": "Reviewer_BNCX", "summary": "This paper addresses the problem of spurious correlations caused by environments from where data are collected.\nThe proposed method applies a mask to input data to separate spurious and semantic features.\nThe masked input data are fed into a local model specialized to each environment.\nEach local model is trained to induce neural collapse for OOD generalization.", "questions": "- Q1: Why do the accuracies of IRM (w/ env) in Tables 1 and 2 differ?\n- Q2: When environment labels are not available, how are spurious and semantic features learned? Is there a possibility that the two types of features are learned conversely?\n- Q3: How do we determine the number of local models when the total number of the environments is unknown?", "limitations": "Discussed in Section 6.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- S1: Making use of neural collapse for OOD generalization is interesting.", "weaknesses": "- W1: Comparison with not only OOD generalization methods but also spurious correlation (sometimes called bias or shortcut) methods is necessary. Methods that can automatically detect and split spurious and semantic features have been developed [a-e].\n- W2: Types of spurious features that the proposed method can handle need to be clarified. Can the proposed method handle spurious features in superposition, e.g., objects and textures?\n- W3: The rationale behind the proposed method needs to be clarified. For instance, it is unclear why the method adds the noise to the mask when learning it.\n- W4: Deeper analyses in the experiments would make the paper more interesting. For example,      \n  - Whether the neural collapse is achieved by the proposed method should be confirmed in the experiment.    \n  - Visualizing learned masks would produce more valuable insights.\n- W5: What is described in the introduction and what is done in the proposed method seems to be different. Although L42 states that `we propose to compute the Frobenius norm (F-norm) of the difference between the feature prototypes and the standard simplex ETF`, the F-norm does not appear in the proposed method.\n- W6: Writing and formatting can be improved. There are many inconsistent spellings. For example,    \n  - Is \"variable features\" in L153 the same as spurious features?  \n  - The meaning of \"interaction\" in L189, 192, and so on is unclear. Maybe \"training?\"  \n  - Such inconsistent spellings occur from Section 4.\n\n[a] Tiwari, Rishabh, and Pradeep Shenoy. \"Overcoming simplicity bias in deep networks using a feature sieve.\" ICML2023.    \n[b] Bahng, Hyojin, et al. \"Learning de-biased representations with biased representations.\" ICML2020.    \n[c] Yang, Wanqian, et al. \"Chroma-vae: Mitigating shortcut learning with generative classifiers.\" NeurIPS2022.    \n[d] Liu, Evan Z., et al. \"Just train twice: Improving group robustness without training group information.\" ICML2021.    \n[e] Nam, Junhyun, et al. \"Learning from failure: De-biasing classifier from biased classifier.\" NeurIPS2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "****Respond to W1:**** Thanks for the comments. We would like to explain as follows:\n\nThe OOD methods encompass techniques for addressing spurious correlations. Our comparison methods are comprehensive, covering both approaches that handle spurious correlations, such as IRM, VREx, GroupDRO, and other OOD methods like DG, including MLDG, MMD, and CDANN. Additionally, we have compared our work with those proposed by the reviewer. Our setting is distinct in that [a-e] address bias issues, while we focus on spurious correlations and follow [1] to benchmark our approach against state-of-the-art baselines.\n\n**Comparison with [a]:** [a] involves utilizing an auxiliary network to identify and eliminate spurious features in the feature space early on. While, our method leverages the phenomenon of neural collapse to achieve feature alignment, thereby eliminating spurious features.\n\n**Comparison with [b]:** [b] utilizes HSIC to achieve independence and learns invariant representations on biased data. In contrast, our method employs a two-stage interactive process, utilizing neural collapse to ultimately achieve learning of invariant features.\n\n**Comparison with [c]:**  [c] achieves invariant representation learning during network training by partitioning the hidden layer space into shortcut and complementary subspaces, guided by loss functions. In contrast, our method automatically partitions across different environments, leveraging the properties of class-invariant features to align class-specific features in various environments.\n\n**Comparison with [d]:**  [d] also follows a two-stage process: the first stage utilizes ERM for training, and the second stage handles instances where ERM identification errors occur. In contrast, our approach first partitions environments. Then, we employ neural collapse to align class-specific features.\n\n**Comparison with [e]:**  [e] involves constructing two networks: one amplifies bias information using GCE, representing variable information, while the other learns debiased information, representing semantic information. In contrast, we employ a two-stage interactive approach to learn invariant networks.\n\n[1] Map: Towards balanced generalization of iid and ood through model-agnostic adapters, 2023.\n\n****Respond to W2:**** Thanks for your questions. We would like to explain this question as follows: We are currently following existing work where neither the baseline models in the literature nor the existing benchmarks incorporate stacked features.\n\n****Respond to W3:**** Thanks for the comments. We would like to clarify it as follows:\nWe follow the approach in [1,2], where we introduce some noise during the initialization of masks to enhance their randomness. This enables the masks to better capture invariant features.\n\n[1] Invariant Representation Learning for Multimedia Recommendation\n[2] Kernelized Heterogeneous Risk Minimization\n\n****Respond to W4:**** Thanks for the comments. We would like to clarify it as follows:\nIn the introduction, we employ the F-norm to assess the feature conditions resulting from different methods. According to the theory of neural collapse, after network training, features collapse onto a simplex. To visually illustrate this phenomenon, we use the F-norm to measure the discrepancy between the features post-training and a standard simplex, as depicted in our first figure.\nIn the main text, we utilize Eq. 5 to align the post-trained features to the standard simplex using a push-pull mechanism. In different environments, invariant features are inherently more similar, thus they consistently align to the same ETH.\n\n****Respond to W5:**** We will incorporate significant improvements in the subsequent revisions of the paper.\n\n****Respond to Q1:**** Thanks for the very constructive comments. We would like to clarify it as follows:\nIn Table 1, we directly apply the IRM algorithm to restrict parameter variations, thereby achieving invariant learning without utilizing a masking mechanism. In Table 2, to demonstrate that our approach with masking can also yield improved results, we substitute the process of learning masks with the IRM loss as shown bellow. \n$$\\mathcal{L}\\_{mask}=\\mathbb{E}\\_{e\\in\\mathcal{E}}\\mathcal{L}^{e}+\\alpha\\\\|\\mathrm{Var}\\_{e\\in\\mathcal{E}}(\\nabla\\_{\\Theta^{mask}}\\mathcal{L}^{e})\\odot\\mu\\\\|^{2}+\\lambda\\\\|\\mathbf{m}\\\\|^{2},$$\nwhere the $\\mathcal{L}^{e}$ is the classification loss, the second term is the constraint across environments, and the third term is a regularization term. $\\mathcal{L}^e$ is the average loss value inside the environment $e$.\nConsequently, the accuracies differ between the two tables.\n\n****Respond to Q2:**** Thanks for the comment. We would like to clarify it as follows:\nIn our paper, we have clarified that when the environment is unknown, we first randomly partition the data into environments. Subsequently, through iterative interaction between environment partitioning as described in Algorithm 1 and mask learning, we achieve convergence in both environment partitioning and mask learning processes. We learn an invariant mask to distinguish between spurious and semantic features in the input data. The learning phase of the invariant mask minimizes the loss function defined in Eq. 5. When this process converges, we assert that the features align well with the standard simplex post-training. Importantly, since we minimize the loss on the masked features, which represent semantic features, there is no issue of conflicting learning between two types of features.\n\n****Respond to Q3:**** Thanks for the comments. We would like to clarify it as follows:\nWhen the number of environments is unknown, we treat it as a hyperparameter in accordance with our previous method. By adjusting the size of this hyperparameter, we aim to achieve optimal out-of-distribution (OOD) generalization performance. In our paper, we also conducted ablation experiments to investigate scenarios where the number of environments is unknown.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 3 reviews from experts in the field. The paper received the following reviews:  1 Accept, 1 Borderline Accept and 1 Borderline Reject.

The main concerns presented by the reviewers included experimentation and clarity. The author rebuttal with additional experiments persuaded the reviewers to increase their ratings. The AC agrees with the points presented by the reviewers and the decision for this paper is to accept. Please include the additional experiments in your paper and address the reviewers’ concerns about clarity in the camera ready version.
Decision: Accept (poster)