SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yongqiang Chen $^{1*}$ , Wei Huang $^{2*}$ , Kaiwen Zhou $^{1*}$
<sup>1</sup>The Chinese University of Hong Kong <sup>2</sup>RIKEN AIP
{yqchen, kwzhou, jcheng}@cse.cuhk.edu.hk wei.huang.vr@riken.jp
Yatao Bian $^{3}$ , Bo Han $^{4}$ , James Cheng $^{1}$
$^{3}$ Tencent AI Lab  $^{4}$ Hong Kong Baptist University
yatao.bian@gmail.com bhanml@comp.hkbu.edu.hk
A common explanation for the failure of out-of-distribution (OOD) generalization is that the model trained with empirical risk minimization (ERM) learns spurious features instead of invariant features. However, several recent studies challenged this explanation and found that deep networks may have already learned sufficiently good features for OOD generalization. Despite the contradictions at first glance, we theoretically show that ERM essentially learns both spurious and invariant features, while ERM tends to learn spurious features faster if the spurious correlation is stronger. Moreover, when fed the ERM learned features to the OOD objectives, the invariant feature learning quality significantly affects the final OOD performance, as OOD objectives rarely learn new features. Therefore, ERM feature learning can be a bottleneck to OOD generalization. To alleviate the reliance, we propose Feature Augmented Training (FeAT), to enforce the model to learn richer features ready for OOD generalization. FeAT iteratively augments the model to learn new features while retaining the already learned features. In each round, the retention and augmentation operations are performed on different subsets of the training data that capture distinct features. Extensive experiments show that FeAT effectively learns richer features thus boosting the performance of various OOD objectives<sup>1</sup>.
Understanding feature learning in neural networks is crucial to understanding how they generalize to different data distributions [2, 11, 12, 62, 67, 70]. Deep networks trained with empirical risk minimization (ERM) learn highly predictive features that generalize surprisingly well to in-distribution (ID) data [24, 75]. However, ERM also tends to learn spurious features or shortcuts such as image backgrounds [7, 19, 23, 84] whose correlations with labels do not hold in the out-of-distribution (OOD) data, and suffers from serious performance degeneration [39]. Therefore, it is widely believed that the reason for the OOD failures of deep networks is that ERM fails to learn the desired features that have invariant correlations with labels across different distributions [7].
However, several recent works find that ERM-trained models have already learned sufficiently good features that are able to generalize to OOD data [33, 38, 63]. In addition, when optimizing various penalty terms [1, 15, 40, 41, 53, 55, 57, 61, 69, 76, 85] that aim to regularize ERM to capture the invariant features (termed as OOD objectives), there also exists a curious phenomenon that the
*Equal Contribution. Work done during Yongqiang's internship at Tencent AI Lab.
$^{1}$ Code is available at https://github.com/LFphase/FeAT.
Figure 1: (a) An illustration of FeAT (top row) compared to ERM (bottom row). Different colors in samples denote the respective dominant features. As the original data is dominated by spurious features (blue), ERM tends to learn more spurious features but limited invariant features (orange). Thus the OOD training with IRMv1 can only leverage limited invariant features and achieve limited performance. In contrast, iteratively, FeAT divides  $\mathcal{D}_{\mathrm{tr}}$  into augmentation  $D^{a}$  and retention sets  $D^{r}$  that contain features not learned and already learned by the current model at the round, respectively. In each round, FeAT augments the model with new features contained in the growing augmentation sets while retaining the already learned features contained in the retention sets, which will lead the model to learn richer features for OOD training and obtain a better OOD performance. Then FeAT augments the model with new features while retaining the already learned features, which leads to richer features for OOD training and better OOD performance. (b) OOD Performance vs. the number of ERM pre-training epochs in COLOREDMNIST-025. The performance of various OOD objectives largely relies on the quality of ERM-learned features. When there exist underlying useful features poorly learned by ERM, the OOD performance will be limited. In contrast, FeAT learns richer features with 2 rounds (or 300 epochs) and improves the OOD performance.
performance of OOD objectives largely relies on the pre-training with ERM before applying the OOD objectives [16, 83]. As shown in Fig. 1(b), the number of ERM pre-training epochs has a large influence on the final OOD performance. These seemingly contradicting phenomena raise a challenging research question:
What features are learned by ERM and OOD objectives, respectively, and how do the learned features generalize to in-distribution and out-of-distribution data?
To answer the question, we conduct a theoretical investigation of feature learning in a two-layer CNN network, when trained with ERM and a widely used OOD objective, IRMv1 [4], respectively. We use a variation of the data models proposed in [2, 12], and include features with different correlation degrees to the labels to simulate invariant and spurious features [36].
First, we find that ERM essentially learns both spurious features and invariant features (Theorem 4.1). The degrees of spurious and invariant feature learning are mostly controlled by their correlation strengths with labels. Moreover, merely training with IRMv1 cannot learn new features (Theorem 4.2). Therefore, the quality of ERM feature learning affects the final OOD performance significantly. Hence, as the number of ERM pre-training epochs increases, the model learns invariant features better and thus the final OOD performance will increase (Fig. 1). However, when ERM does not capture all useful features for OOD generalization, i.e., there exist some useful features that are poorly learned by ERM, the model can hardly learn these features during OOD training and the OOD performance will be limited. Given a limited number of pre-training steps, it could often happen due to low invariant correlation strength, the feature learning biases of ERM [67], or the model architectures [28]. Consequently, ERM feature learning can be a bottleneck to OOD generalization [60].
To remedy the issue, we propose Feature Augmented Training (FeAT), an iterative strategy to enforce the model to learn richer features. As shown in Fig. 1(a), in each round, FeAT separates the train set into two subsets according to whether the underlying features in each set are already learned (Retention set  $\mathcal{D}^r$ ) or not (Augmentation set  $\mathcal{D}^a$ ), by examining whether the model yields correct  $(\mathcal{D}^r)$  or incorrect  $(\mathcal{D}^a)$  predictions for samples from the subsets, respectively. Intuitively,  $\mathcal{D}^a$  and  $\mathcal{D}^r$  will contain distinct features that are separated in different rounds. Then, FeAT performs distributionally
robust optimization (DRO) [50, 83] on all subsets, which augments the model to learn new features by minimizing the maximal ERM losses on all  $\mathcal{D}^a$  and retains the already learned features by minimizing ERM losses on all  $\mathcal{D}^r$ . Along with the growth of the augmentation and retention sets, FeAT is able to learn richer features for OOD training and obtain a better OOD performance. FeAT terminates when the model cannot learn any new predictive features (Algorithm 1).
We conduct extensive experiments on both COLOREDMNIST [4, 16] and 6 datasets from the challenging benchmark, WILDS [39], and show that FeAT effectively learns richer features and thus consistently improves the OOD performance when applied to various OOD objectives (Sec. 6).
We discuss the most related work to ours and leave more details in Appendix C.
On Feature Learning and Generalization. Understanding feature learning in deep networks is crucial to understanding their generalization [2, 11, 12, 22, 32, 62, 70]. Beyond the empirical probing [21, 26, 28, 65], Allen-Zhu and Li [2] proposed a new theoretical framework for analyzing the feature learning process of deep networks, which has been widely adopted to study various deep learning phenomena [12, 32, 78, 86]. However, how the learned features from ID data can generalize to OOD data remains elusive. The only exceptions are [68] and [42]. Kumar et al. [42] find fine-tuning can distort the pre-trained features while fine-tuning can be considered as a special case in our framework. Shen et al. [68] focus on how data augmentation helps promote good but hard-to-learn features and improve OOD generalization. Deng et al. [20] finds neural networks tend to learn spurious features under imbalanced groups. In contrast, we study the direct effects of ERM and OOD objectives to feature learning and provide a theoretical explanation for the curious phenomenon [33, 63]. To the best of our knowledge, we are the first to analyze the feature learning of ERM and OOD objectives and their interactions in the general OOD generalization setting.
Rich Feature Learning. Recently many OOD objectives have been proposed to regularize ERM such that the model can focus on learning invariant features [4, 41, 55, 57, 76]. However, the final OOD performance has a large dependence on the number of ERM pre-training epochs [16, 83]. To remedy the issue, Zhang et al. [83] proposed Bonsai to construct rich feature representations as network initialization for OOD training. Although both Bonsai and FeAT perform DRO on grouped subsets, Bonsai rely on multiple initializations of the whole network to capture diverse features from the subsets, and complicated assembling of the features, which requires more training epochs for convergence. In contrast, FeAT relieves the requirements via direct augmentation-retention on the grouped subsets, and thus obtains better performance. More crucially, although rich feature learning algorithms such as Bonsai and weight averaging [5, 59] have gained some successes, explanations about the reliance of OOD performance on ERM pre-training and why rich feature learning mitigates the issue remain elusive. In addition to a new rich feature learning algorithm, our work provides theoretical explanations for the success of rich feature learning in OOD generalization.
3 Preliminaries and Problem Definition
Notations. We use old-faced letters for vectors and matrices otherwise for scalar;  $\| \cdot \| _2$  to denote the Euclidean norm of a vector or the spectral norm of a matrix, while  $\| \cdot \| _F$  for the Frobenius norm of a matrix.  $\mathbf{I}_d$  refers to the identity matrix in  $\mathbb{R}^{d\times d}$ . Full details are deferred to Appendix A.
Our data model  $\mathcal{D} = \{\mathbf{x}_i, y_i\}_{i=1}^n$  is adapted from [2, 12] and further characterizes each data point  $\mathbf{x}_i$  as invariant and spurious feature patches from the two-bit model [16, 36].
Definition 3.1.  $\mathcal{D} = \{\mathcal{D}_e\}_{e\in \mathcal{E}_{\mathrm{all}}}$  is composed of multiple subsets  $\mathcal{D}_e$  from different environments  $e\in \mathcal{E}_{\mathrm{all}}$ , where each  $\mathcal{D}_e = \{(\mathbf{x}_i^e,y_i^e)\}_{i = 1}^{n_e}$  is composed of i.i.d. samples  $(\mathbf{x}_i^e,y_i^e)\sim \mathbb{P}^e$ . Each data  $(\mathbf{x}^e,y^e)\in \mathcal{D}_e$  with  $\mathbf{x}^e\in \mathbb{R}^{2d}$  and  $y^{e}\in \{-1,1\}$  is generated as follows:
(a) Sample  $y^{e}\in \{-1,1\}$  uniformly;
(b) Given  $y^{e}$ , each input  $\mathbf{x}^{e} = [\mathbf{x}_{1}^{e},\mathbf{x}_{2}^{e}]$  contains a feature patch  $\mathbf{x}_1$  and a noise patch  $\mathbf{x}_2$ , that are sampled as:
$$
\mathbf {x} _ {1} = y \cdot \operatorname {R a d} (\alpha) \cdot \mathbf {v} _ {1} + y \cdot \operatorname {R a d} (\beta) \cdot \mathbf {v} _ {2} \quad \mathbf {x} _ {2} = \boldsymbol {\xi}
$$
where  $\operatorname{Rad}(\delta)$  is a random variable taking value  $-1$  with probability  $\delta$  and  $+1$  with probability  $1 - \delta$ ,  $\mathbf{v}_1 = [1,0,\ldots,0]^\top$  and  $\mathbf{v}_2 = [0,1,0,\ldots,0]^\top$ .
(c) A noise vector  $\pmb{\xi}$  is generated from the Gaussian distribution  $\mathcal{N}(\mathbf{0},\sigma_p^2\cdot (\mathbf{I}_d - \mathbf{v}_1\mathbf{v}_1^\top -\mathbf{v}_2\mathbf{v}_2^\top))$
Definition 3.1 is inspired by the structure of image data in image classification with CNN [2], where the inputs consist of different patches, some of the patches consist of features that are related to the class label of the image, and the others are noises that are irrelevant to the label. In particular,  $\mathbf{v}_1$  and  $\mathbf{v}_2$  are feature vectors that simulate the invariant and spurious features, respectively. Although our data model focuses on two feature vectors, the discussion and results can be further generalized to multiple invariant and spurious features with fine-grained characteristics [68]. Following previous works [12], we assume that the noise patch is generated from the Gaussian distribution such that the noise vector is orthogonal to the signal vector  $\mathbf{v}$ . Each environment is denoted as  $\mathcal{E}_{\alpha} = \{(\alpha ,\beta_{e}):0 < \beta_{e} < 1\}$ , where  $\mathbf{v}_1$  is the invariant feature as  $\alpha$  is fixed while  $\mathbf{v}_2$  is the spurious feature as  $\beta_{e}$  varies across  $e$ .
CNN model. We consider training a two-layer convolutional neural network with a hidden layer width of  $m$ . The filters are applied to  $\mathbf{x}_1, \mathbf{x}_2$ , respectively, and the second layer parameters of the network are fixed as  $\frac{1}{m}$  and  $-\frac{1}{m}$ , respectively. Then the network can be written as  $f(\mathbf{W}, \mathbf{x}) = F_{+1}(\mathbf{W}_{+1}, \mathbf{x}) - F_{-1}(\mathbf{W}_{-1}, \mathbf{x})$ , where  $F_{+1}(\mathbf{W}_{+1}, \mathbf{x})$  and  $F_{-1}(\mathbf{W}_{-1}, \mathbf{x})$  are defined as follows:
$$
F _ {j} \left(\mathbf {W} _ {j}, \mathbf {x}\right) = \frac {1}{m} \sum_ {r = 1} ^ {m} \left[ \psi \left(\mathbf {w} _ {j, r} ^ {\top} \mathbf {x} _ {1}\right) + \psi \left(\mathbf {w} _ {j, r} ^ {\top} \mathbf {x} _ {2}\right) \right], \tag {1}
$$
where  $\psi (x)$  is the activation function. We assume that all network weights are initialized as  $\mathcal{N}(0,\sigma_0^2)$ .
ERM objective. We train the CNN model by minimizing the empirical cross-entropy loss function:
$$
L (\mathbf {W}) = \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell \left(y _ {i} ^ {e} \cdot f \left(\mathbf {W}, \mathbf {x} _ {i} ^ {e}\right)\right), \tag {2}
$$
where  $\ell(z) = \log(1 + \exp(-z))$  and  $\{\mathcal{D}_e\}_{e \in \mathcal{E}_{\mathrm{tr}}} = \{\{\mathbf{x}_i^e, y_i^e\}_{i=1}^{n_e}\}_{e \in \mathcal{E}_{\mathrm{tr}}}$  is the trainset with  $\sum_{e \in \mathcal{E}_{\mathrm{tr}}} n_e = n$ .
OOD objective. The goal of OOD generalization is, given the data from training environments  $\{\mathcal{D}_e\}_{e\in \mathcal{E}_{\mathrm{tr}}}$ , to find a predictor  $f:\mathcal{X}\to \mathcal{Y}$  that generalizes well to all (unseen) environments, or minimizes  $\max_{e\in \mathcal{E}_{\mathrm{all}}}L_e(f)$ , where  $L_{e}$  is the empirical risk under environment  $e$ . The predictor  $f = w\circ \varphi$  is usually composed of a featurizer  $\varphi :\mathcal{X}\rightarrow \mathcal{Z}$  that learns to extract useful features, and a classifier  $w:\mathcal{Z}\rightarrow \mathcal{V}$  that makes predictions from the extracted features.
Since we are interested in cases where the OOD objective succeeds in learning the invariant features. In the discussion below, without loss of generality, we study one of the most widely discussed OOD objective, IRMv1 objective, from IRM framework [4], and the data model where IRMv1 succeeds. Specifically, the IRM framework approaches OOD generalization by finding an invariant representation  $\varphi$ , such that there exists a classifier acting on  $\varphi$  that is simultaneously optimal in  $\mathcal{E}_{\mathrm{tr}}$ . Hence, IRM leads to a challenging bi-level optimization problem as
$$
\min  _ {w, \varphi} \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} L _ {e} (w \circ \varphi), \text {s . t .} w \in \underset {\bar {w}: \mathcal {Z} \rightarrow \mathcal {Y}} {\arg \min } L _ {e} (\bar {w} \circ \varphi), \forall e \in \mathcal {E} _ {\mathrm {t r}}. \tag {3}
$$
Due to the optimization difficulty of Eq. (3), Arjovsky et al. [4] relax Eq. (3) into IRMv1 as follows:
$$
\min  _ {\varphi} \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} L _ {e} (\varphi) + \lambda | \nabla_ {w | w = 1} L _ {e} (w \cdot \varphi) | ^ {2}. \tag {4}
$$
Given the convolutional neural network (Eq. 1) and logistic loss (Eq. 2), IRMv1 can be written as
$$
L _ {\mathrm {I R M v 1}} (\mathbf {W}) = \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell \left(y _ {i} ^ {e} \cdot f \left(\mathbf {W}, \mathbf {x} _ {i} ^ {e}\right)\right) + \sum_ {e \in \mathcal {E} _ {\mathrm {t r}}} \frac {\lambda}{n _ {e} ^ {2}} \left(\sum_ {i = 1} ^ {n _ {e}} \ell_ {i} ^ {\prime e} \cdot y _ {i} ^ {e} \cdot f \left(\mathbf {W}, \mathbf {x} _ {i} ^ {e}\right)\right) ^ {2}, \tag {5}
$$
where  $\ell_i^{\prime e} = \ell^{\prime}(y_i^e\cdot f(\mathbf{W},\mathbf{x}_i^e)) = -\frac{\exp(-y_i^e\cdot f(\mathbf{W},\mathbf{x}_i^e))}{1 + \exp(-y_i^e\cdot f(\mathbf{W},\mathbf{x}_i^e))}$ . Due to the complexity of IRMv1, in the analysis below, we introduce  $C_{\mathrm{IRMv1}}^{e}$  for the ease of expressions. Specifically, we define  $C_{\mathrm{IRMv1}}^{e}$  as
$$
C _ {\mathrm {I R M v 1}} ^ {e} \triangleq \frac {1}{n _ {e}} \sum_ {i = 1} ^ {n _ {e}} \ell^ {\prime} \left(y _ {i} ^ {e} \hat {y} _ {i} ^ {e}\right) \cdot y _ {i} ^ {e} \hat {y} _ {i} ^ {e},
$$
where  $\hat{y}_i^e\triangleq f(\mathbf{W},\mathbf{x}_i^e)$  is the logit of sample  $\mathbf{x}_i$  from environment  $e$ . The convergence of  $C_{\mathrm{IRMv1}}$  indicates the convergence of IRMv1 penalty. The following lemma will be useful in our analysis.
2When the environment  $e$  is not explicitly considered, we will omit it for clarity.
(a)  $C_{\mathrm{IRMV1}}$  ,w/PT
(b) FL, w/ PT
Figure 2: The convergences of  $C_{\mathrm{IRMv1}}$  and feature learning coefficients (FL) with or without ERM pre-training (PT). The invariant and spurious feature learning terms are the mean of  $\langle \mathbf{w}_{j,r}, j\mathbf{v}_1 \rangle$  and  $\langle \mathbf{w}_{j,r}, j\mathbf{v}_2 \rangle$  for  $j \in \{\pm 1\}$ ,  $r \in [m]$ , respectively. The training environments are  $\mathcal{E}_{tr} = \{(0.25, 0.1), (0.25, 0.2)\}$ . The black dashed line indicates the end of pre-training. More details are given in Appendix D.1.
(c)  $C_{\mathrm{IRMv1}}$  ,w/o PT
(d) FL, w/o PT
Lemma 3.2. (Cao et al. [12]) Let  $\mathbf{w}_{j,r}(t)^3$  for  $j\in \{+1, - 1\}$  and  $r\in \{1,2,\dots ,m\}$  be the convolution filters of the CNN at t-th iteration of gradient descent. Then there exists unique coefficients  $\gamma_{j,r}^{inv}(t),\gamma_{j,r}^{spu}(t)\geq 0$  and  $\rho_{j,r,i}(t)$  such that,
$$
\mathbf {w} _ {j, r} (t) = \mathbf {w} _ {j, r} (0) + j \cdot \gamma_ {j, r} ^ {i n v} (t) \cdot \mathbf {v} _ {1} + j \cdot \gamma_ {j, r} ^ {s p u} (t) \cdot \mathbf {v} _ {2} + \sum_ {i = 1} ^ {n} \rho_ {j, r, i} (t) \cdot \| \boldsymbol {\xi} _ {i} \| _ {2} ^ {- 2} \cdot \boldsymbol {\xi} _ {i}. \tag {6}
$$
We refer Eq. (6) as the signal-noise decomposition of  $\mathbf{w}_{j,r}(t)$  [12]. We add normalization factor  $\| \pmb{\xi}_i\|_2^{-2}$  in the definition so that  $\rho_{j,r}^{(t)} \approx \langle \mathbf{w}_{j,r}^{(t)}, \pmb{\xi}_i \rangle$ . Note that  $\| \mathbf{v}_1\|_2 = \| \mathbf{v}_2\|_2 = 1$ , the corresponding normalization factors are thus neglected. Furthermore,  $\gamma_{j,r}^{inv} \approx \langle \mathbf{w}_{j,r}, \mathbf{v}_1 \rangle$  and  $\gamma_{j,r}^{spu} \approx \langle \mathbf{w}_{j,r}, \mathbf{v}_2 \rangle$  respectively denote the degrees of invariant and spurious feature learning.
4 Theoretical Understanding of Feature Learning in OOD Generalization
4.1 ERM Feature Learning
With the setup in Sec. 3, we first study the feature learning of the ERM objective. We consider a two training environments setup  $\mathcal{E}_{tr} = \{(\alpha, \beta_1), (\alpha, \beta_2)\}$  where the signal of invariant feature is weaker than the average of spurious signals (i.e.,  $\alpha > \frac{\beta_1 + \beta_2}{2}$ ), which corresponds to Figure 2. For a precise characterization of the training dynamic, we adopted a minimal setup where  $\psi(x) = x$  in Figure 2(a) and the following theorem, which already captures the key phenomenon in ERM feature learning. We study ERM feature learning with non-linear activations in Appendix D.2.3.
Theorem 4.1. (Informal) For  $\rho >0$ , let  $\underline{n}\triangleq \min_{e\in \mathcal{E}_{tr}}n_e$ . Suppose that we run  $T$  iterations of  $GD$  for the ERM objective. With sufficiently large  $\underline{n}$  and  $\psi (x) = x$ , assuming that (i)  $\alpha ,\beta_{1},\beta_{2} < \frac{1}{2}$ , and (ii)  $\alpha >\frac{\beta_1 + \beta_2}{2}$ , with properly chosen  $\sigma_0^2$  and  $\sigma_p^2$ , there exists a constant  $\eta$ , such that with probability at least  $1 - 2\rho$ , both invariant and spurious features are converging and the increment of the spurious feature is larger than that of the invariant feature at any iteration  $t\in \{0,\dots,T - 1\}$  (the detailed quantitative result of this gap can be found at (15) in Appendix D.2).
As the formal statement of Theorem 4.1 is too complicated and lengthy, we leave it and its proof in Appendix D.2, while giving an informal but more intuitive version here. Theorem 4.1 states that ERM training learns both invariant feature and spurious feature at the same time, and if the average of spurious signals is stronger, the coefficient of spurious feature learning will dominate that of invariant feature learning in the whole training process, corresponding to Figure 2(b). We establish the proof based on inspecting a novel recursive equation, which might be of independent interest. Note that Theorem 4.1 can be directly generalized to handle any number of environments.
Speaking of implications, Theorem 4.1 provides answers to the seemingly contradicting phenomena that ERM fails in OOD generalization [7, 19] but still learns the invariant features [33, 38, 63]. In fact, ERM fails since it learns the spurious features more quickly, when spurious correlations are stronger than invariant correlations. Nevertheless, invariant feature learning also happens, even when the spurious correlations are strong, so long as the invariant feature has a non-trivial correlation strength with the labels. Therefore, simply re-training a classifier based on a subset of unbiased data on top of the ERM-trained featurizer achieves impressive OOD generalization performance [33, 38, 63].
3We use  $\mathbf{w}_{j,r}(t),\mathbf{w}_{j,r}^{(t)}$  and  $\mathbf{w}_{j,r}^t$  interchangeably.
Theorem 4.1 also provides an explanation for the ID-OOD performance correlations when fine-tuning or training neural networks (especially large pre-trained models like CLIP [56], GPT [10]) [46, 71, 79, 80]. We provide a detailed discussion in Appendix C.
4.2 IRM Feature Learning
Although Theorem 4.1 states that ERM learns both invariant and spurious features, the following questions remain unanswered: (1) whether IRMv1 learns new features or simply amplifies the already learned ERM features, and (2) how the quality of the ERM-learned features affects the feature learning when IRMv1 is incorporated. We first study IRMv1 training from scratch (w/o pre-training).