Intermediate Layer Classifiers for OOD generalization
Authors: Arnas Uselis
Seong Joon Oh
Venue: ICLR 2025 Poster (2025)
Keywords: transfer learning
intermediate layers
learning dynamics
OOD generalization
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Deep classifiers are known to be sensitive to data distribution shifts, primarily due to their reliance on spurious correlations in training data. It has been suggested that these classifiers can still find useful features in the network's last layer that hold up under such shifts. In this work, we question the use of last-layer representations for out-of-distribution (OOD) generalisation and explore the utility of intermediate layers. To this end, we introduce \textit{Intermediate Layer Classifiers} (ILCs). We discover that intermediate layer representations frequently offer substantially better generalisation than those from the penultimate layer. In many cases, zero-shot OOD generalisation using earlier-layer representations approaches the few-shot performance of retraining on penultimate layer representations. This is confirmed across multiple datasets, architectures, and types of distribution shifts. Our analysis suggests that intermediate layers are less sensitive to distribution shifts compared to the penultimate layer. These findings highlight the importance of understanding how information is distributed across network layers and its role in OOD generalisation, while also pointing to the limits of penultimate layer representation utility. Code is available at https://github.com/oshapio/intermediate-layer-generalization.

Published as a conference paper at ICLR 2025
INTERMEDIATE LAYER CLASSIFIERS FOR OOD GENER- ALIZATION
Arnas Uselis
Scong Joon Oh Tübingen Al Center University of Tübingen
Tübingen AI Center
University of Tübingen a arnas usel ssuuni issuni--tuebe -tweiiinnttebiinnnnn.nnn de
ABSTRACT
Deep classifiers are known to be sensitive to data distribution shifts, primarily due their reliance on spurious correlations in training data. has been suggested that these classifiers can still find useful features in the network's 's last layer that hold up under such shifts. In this work, we question the use of last-layer rep- resentations for out-of-distribution (OOD) generalisation and explore the utility intermediate layers. To this end, we introduce Intermediate Layer Classifiers (ILCs). We discover that intermediate layer representations frequently offer sub- stantially better generalisation than those from the penultimate layer. In many cases, zero-shot OOD generalisation using carlier-layer representations approaches the few-shot performance of retraining on penultimate layer representations. This is confirmed across multiple datasets, architectures, and types of distribution shifts. Our analysis suggests that intermediate layers are less sensitive to distribution shifts compared the penultimate layer. These findings highlight the importance of understanding how information is distributed across network layers and its role in OOD generalisation, while also pointing to the limits of penultimate layer representation utility. Code is available at https::: //gtth/gtthtbt.. com/ oshapio/
intermedi ate- layer genereneralle
CMNIST
CIFAR1O-C
Waterbirds
CelebA
100%
+-0%
+12%
+0%
+0%
+7%
+0%
+0%
+0%
Accuracy
+25%
+6%
75%
+7%
+3%
50%
ID
kero-shotFew-shot OOD
ID Zero-shotFew-shot OOD
ID Zero-shotFew-shet OOD OOD
ID Zero-shotFew-shot 0OD OOD
OOD
0OD
Figure 1: Using last vs intermediate layers for OOD generalisation. A common way address distribution shift is to fine-tune the layer of network on the target distribution (few-shot learning). show that earlier- layer representations often generalise better than the last layer. Moreover, even when only the in-distribution (ID) data available, earlier-layer representations are often better than the last layer (zero-shot learning).
INTRODUCTION
Deep neural networks (DNNs) often lack robustness when evaluated on out-o--disttibution (OOD) samples: once a classifier is trained, its performance often drops significantly when deployed in new environment (Taori et 2020).
reason for this is that training data often contain spurious
correlations (Singla la & Feizi| 2021 Li et al. 2023), which encourage models to learn shortcuts
(Scimeca al.] Cadene et al. 2020 2020, Recht al. 2019). Relying on such shortcuts leads to models that do not generalize well to out-of-distribution (OOD) samples (Geirhos et al. 2020 2022 Rosenteld et al. 2018 Beery et al. 2018) lying outside the training distribution. Many attempts have been made address the disparity in performance between in in-distribution (ID) and OOD samples (Zhang et al. 20118 Yun et al. 2019 Shi et al. [002 Verma et al. 2019)), but when no testing data is available from the target distribution, generalization is challenging.

---

Published as a conference paper at ICLR 2025
Recent work has shown that the last layer of DNNs already contain enough information for general- ization in the cases of long-tail classification (Kang et al. 2020), domain generalization (Rosenfeld et al., 2022). and learning under spurious correlations (Kirichenko et al. (023). In these works, only the linear classifier of the last layer is retrained on the target distribution, and the model is shown to generalize well to OOD samples. This suggests that the model can learn useful representations from the training data alone already, and only the classifier needs to be adjusted for the target distribution.
This work questions the conventional usage of the last layer for OOD generalization, providing an in-depth analysis across various distribution shifts and model architectures. We examine both few-shot OOD generalization, where some OOD samples are available for training linear classifier, and zero-shot OOD generalization, which requires no target examples. Our studies reveal that earlier- layer representations often yield superior OOD generalization in both scenarios. Figure [ Iillustrates this across multiple datasets. For instance, on CMNIST, using earlier layers improves zero-shot OOD) accuracy by 7% and few-shot OOD accuracy by 12% compared to retraining the last layer. The effect is even more pronounced for CelebA, where we observe substantial gains of 20% for zero-shot OOD generalization using earlier layers. These results consistently demonstrate the advantages of using earlier-layer representations for OOD generalization tasks.
We advocate the use of carlier-layer representations for a few critical benefits in practice. First, we show that they often cxhibit better generalization performance when tuned on the target distribution; this also extends to cases where the number of samples from the target distribution is small ($4.2). Second, the benefits remain even when OOD data is used for training the probes, only for using them in the model-selection step ($4.3).
Our contributions are as follows: (1) We establish that earlier-layer representations often outperform last-layer features in terms of (1) few-shot and (2) zeros-shot transfer capabilities, even when no OOD supervision is available. (3) We provide evidence that intermediate layer features are generally less sensitive to distribution shifts than those from the final layer, offering new insights into feature utility for enhancing model robustness.
2 RELATED WORK
OOD generalization and spurious correlations: It has been reported that spurious correlations in training data degrade the generalizability of learned representations, particularly on the out-of- distribution (OOD) data (Arjovsky et al., [2200 Ruan et al.) 2022 [eermann & Lampinen [2020). especially on groups within the data that are underrepresented (Sagawa et al. 2020aa| Yang et al.)
2023) and the number of groups large (Li et al. 2023] Kim et al. 2023). The theory of gradient starvation further suggests that standard SGD training may preferentialyy leverage spurious correla- tions (Pezeshki et al., 2021), especially when the spurious cue is simple (Scimeca et al. (0022). Other works have focused on the conceptual ill- poseddesssss of OOD generalization without any information
about the target distribution (Ruan et al. 2022 Bahng et al. 2020 [Scimeca et al. 2022).
Simplicity bias, where networks favor easy-to-learn features, is influenced by the breadth of solutions exploiting such features compared to those utilizing more complex signals (Geirhos et al. 2020 Valle-Pérez et al.] 2018, Scimeca al. 2022). It has been observed that DNN models tend to perform well on average but worse for infrequent groups within the data (Sagawa et al. 2020a ; this
especially exacerbated for overparameterized models (Sagawa al., 2020b Menon et al. 2020).
Last layer-retraining: Recent studies have argued that last-layer representations already contain valuable information for generalization beyond the training distribution (Kirichenko et 2023
Izmailov et al. 2022 LaBonte et al., 2025). These approaches suggest retraining the last layer weights (using the features from the penultimate layer) with either target domain data or using group annotations from large language models (Park et al., (003). In this work, we challenge the underlying assumption that the penultimate layer encapsulates all pertinent information for OOD generalization.
Our analysis indicates that earlier-layer representations are often far more useful for OOD generaliza- tion; we even show that earlier-layer representations without fine-tuning on the target distribution often fare competitively with the last-layer retraining on the target distribution.
Exploiting and analyzing intermediate layers. Intermediate layers have been employed for various purposes, from predicting generalization gaps (Jiang et al. 2019), to elucidating training dynamics
2

---

Published as a conference paper at ICLR 2025
(Alain & Bengio 2018), to enhancing transfer and few-shot learning (Evci et al. 2022 Adler et al. 2021). to enhancing transferablity of adversarial examples (Huang et al. 2020). and adjusting models based on distribution shifts (Lee al., 2023). The effectiveness of intermediate layers can be connected to the properties of neural network dynamics Yosinski et al. (014). For example, the intrinsic dimensionality of intermediate layers has been shown to increase in the earlier layers and then decrease in the later layers (Ansuini et al., 2019 Recanatesi et al. 2019). suggesting rich and diverse set of features in the intermediate layers that can be leveraged for generalization. Neural collapse (Papyan et al. 2020), phenomenon where the representations of class features collapse to their mean classes in the last layer, points to a reason for the difficulty of transferring features from the last layer. Neural collapse in intermediate layers is observed, but it is less severe (Li et al. 2022
Rangamani et al. 2023). Recent work has also explored the utility of intermediate layers in distinct contexts, such as generalization to new class splits Gerritz et al. 2024); Dyballa et al. (2024), and has arrived at conclusions similar to ours: that the last layer does not always generalize best. Additionally, Masarczyk et al. (2023) observe a complementary phenomenon termed the tunnel effect, in which later layers compress linearly separable representations created by initial layers, thus degrading OOD performance.
Our work differs by examining distribution shifts within individual datasets, rather than generalization across datasets or tasks. Unlike prior studies, we investigate how intermediate layers generalize under controlled in-dataset shifts, even when training and testing involve similar visual variations.
3
APPROACH
This section introduces the preliminary concepts and notation as well as our approach to OOD) generalization using intermediate-layer representations.
3.1 TASK
We consider the classification task of mapping inputs X' labels y: We present overview of the training and evaluation stages of the model in Table D A deep neural network (DNN) model is first trained on a train- ing dataset Dirain- The DNN is fur- ther adapted to the task through the Intermediate Layer Classifiers (ILCs) training ($3.2) on the probe-training dataset D Pprobe- Afterwards, any model selection or hyperparameter search performed over the validation set Dvalid. Finally, the model is evaluated on the test set Drest-
DNN train
ILC
Valid
Test
Few-shot ($4.
Dusin
Ducke
Dvatid
Dres
Zero-shot ($4.3
Dinin
Dreoke
Distribution
PtD
POOD
Table 1: Experimental setups. We consider settings where an entire deep neural network (DNN) is first trained Duain and one of its layers adapted further on D Prooee (Intermediate Layer Classifiers; ILC in $32).. The model is then validated on Doalid and evaluated on Dest. Depending on the scenario type (few-shot vs zero-shot), either up to Dimin up to Dymbo is considered to be in-distribution PiD and the rest to he out-of-distribution PoOD-
In order to simulate the OOD generalization scenario, we introduce two different distributions: PiD and PooD, respectively denoting in-distribution (ID) and OOD cases. The training of any DNN is performed over ID: Dirai ~ PID. The validation is performed over OOD: D ~ POOD. The usage of a few OOD samples for validation is standard practice in OOD generalization literature (Gulrajani & Lopez-Paz 2020, Sagawa et al.] 2020aa Izmailov et al., 2022) and we adopt this framework for a fair comparison. Likewise, the final multiple variants of OOD datasets; we discuss them in greater detail in $4.1
is performed on OOD: D POoD. We consider
For the ILC training step, consider two possibilities: (1) few-shot where 2 Ppmbee Poon are K OOD samples per class and (2) zero-shot where Drrhhe ~ PID are trained over the ID) set. The last- layer retraining framework (Kirichenko et al. 2023) corresponds to the few-shot learning scenario, where the last layer is retrained with K OOD samples per class. Our research question for the few-shot scenario is whether intermediate representation yields better OOD generalization compared to the last-layer retraining paradigm. For the zero-shot scenario, we venture into more audacious research question: is it possible to train an intermediate representation with ID samples to let generalize to OOD cases?
3

---

Published as a conference paper at ICLR 2025
3.2 INTERMEDIATE LAYER CLASSIFIERS (ILCs)
In this section, we propose the framework for training Intermediate Layer Classifiers (ILC). We start with the necessary notations and background materials.
Let function f be a deep neural network (DNN) classifier with L layers. We denote the I-th layer operation as ft. such that f= fr o fL-1 o ... f1, composition of functions. The last layer of the network f is linear classifier. We refer to the output of the (-th layer for an input x as the /-th layer representation, denoted as (x) == (fi
fx)(x) E R&, where d, denotes the output dimension at layer I. For / < L, we refer to fi as an intermediate layer and ry(x) as an intermediate representation.
@ Pre-trained model (e.g. ResNet or VIT)
Input image
fr trr
Non-blond fa [0.8,
fx(x)
f2(rs)
fa(r2)
ILC
Blond o
W Wir.(x) 00.1,
x
F1 (x)
(x)
rx(
r2 T
Intermediate representations
Figure 2: Intermediate Layer Classifiers (ILC). Given frozen pre- pre-trained model like ResNet a VIT, train a linear probe on an intermediate layer representation at intermediate layers (here, we show this process only at layer l). The composition of 1 layer feature extractor and the intermediate layer classifier (II.C) is the final classifier. We shorthand rx(x) as ry for brevity.
We primarily use ResNets (He al. 2015) for convolutional neural networks and vision transformers (ViTs) (Dosovitskiy et al. 2021) for non-CNN architectures, while other architectures are also used when open-source implementations are available. ResNet layer, for example, consists of convolutions, ReLU activation, batch normalization, and residual connections, leading to L 8 layers in total, excluding the classification head. A ViT layer is an encoder block composed of multi-head attention (MHA) and multi- layer perceptron (MLP). The number of layers for ViT models varies depending on the dataset used. All the architectures used are detailed in Appendix A.
Intermediate Layer Classifiers (ILCs). We introduce Intermediate Layer Classifiers (II.C) to address the spe- eific challenge of OOD tasks. While traditional linear probes (Alain & Bengio 2018) are typically used to ana- lyze learned representations across model, ILCs serve different purpose: they are applied to intermediate layers and trained specifically to perform OOD classification. An ILC at layer is an affine transformation that maps the representation space Re to logits:
Algorithm Training Intermediate Layer Classifiers (IICs)
1: 1 1 L do
2: Initialize weights W; and biases b
for ILC.
3: end for
4: for (x. B) ~ D pmbe do
5: for <
2 do
6:
rı(x) = (fio a ..... fx)(()
IxC;(x) Wir(xx) + bi,
(1)
7:
ýi = ILCi(xx) = Win(()+ end for
8:
where W, E RDix and by IR!* The ILCs for 1 < I < L 1 are trained on the dataset 2 probe ($3.1), using data drawn from ID for the zero-shot scenario (D) probe
9:
>4-1 CE().,))
10: Update parameters & using
11:
end for
PID) and OOD for the few-shot scenario (Dprobe POOD).
We illustrate the data flow of using ILCs conceptually in Figure Last-layer retraining methods (Kirichenko al. 2023 Izmailov et al.] [022] LaBonte et al.] 2023 Kang al. 2020) can be considered special case of the II.C framework, where only the final layer 7. is adapted on D Dprohe for OOD tasks, corresponding to using II.C,,-1 1 in our framework. Algorithm illustraates the training process of ILCs.
Layer Selection. We choose the layer /* with the best ICL accuracy on the validation set Dvalid from 2. 2.
I L last-layer retraining approach. The best layer is chosen based on the best-pero hyperparameters in the search space 7 (detailed in Appendix A.2.2 for each layer. Using few OOD samples for making design choices is one of the common practices in benchmarking OOD generalization (Kirichenko et al. 2023
We restrict the layer selection to layers up to L - to distinguish the results from the
Sagawa al. 2020a).
4

---

Published as a conference paper at ICLR 2025
Inference. For the selected layer 1* Σ L the whole model is reduced to smaller network with 1* layers. Pseudocode for the inference process is provided in Algorithm in the Appendix.
EXPERIMENTS
In this section, we verify the effectiveness of ILCs in $3.2 for out-of-distribution (OOD) generalization. In particular, we compare their performance against the popular last-layer retraining approach. We introduce the dataset and experimental setups in $4.1 We report results under two scenarios: few-shot ($4.2) and zero-shot ($4.3) cases, respectively referring to the availability and unavailability OOD data for the ILC training.
4.1 DATASETS AND EXPERIMENTAL SETUP
We introduce datasets and a precise experimental setup for simulating and studying OOD generaliza- tion. As introduced in the task section ( $311 we need two distributions (X, Y) # Poon for the study. We perform an extensive study over 9 datasets, covering various scenarios, including subpopulation shifts, conditional shifts, noise-level perturbations, and natural image shifts, Detailed definitions of shift types and corresponding datasets are listed in Table 2 below.
Table 2: Distribution shift types and datasets. Datasets were selected based on the availability of distribution shifts and their compatibility with publicly available pre- trairraned model weights. Shift type
Description
Datasets
Conditional
The conditional distribution PX) shifts:
CMNIST (Arjovsky et al.
2020 Bahng et al.
Punin (Y (X) & (Y)
2020
Subpopulation
Given multiple groups within a population. the ratios groups change across distribu- tion.
CelebA Liu al. 2015..
Waterbirds Sagawa
2020a
Multi CelebA
Kim et al 2023
Input noise
Different types of input noise are applied for test samples.
CIFAR-100 CIFAR-100C
Hendryycks & Diet-
terich 2019
Natural image
images have different styles from training images.
ImageNet-A. Conflict, ImageNet-Silhouse Silhouete (Hendryoks et al.]
ImageNet-R,
ImageNet-Cue
2021b a Geirhos et al. 2022
In $31 we have introduced the few-shot and zero-shot settings for the OOD generalization. Below, we explain how we adopt each dataset for the required data splits, Dusin. D probee Dvalid, and Diest-
Detailed information on the datasets and their splits is provided in Appendix A.2
Training split Dirain. For all datasets, we assume DNN models were trained on the given training split.
Probe-training split D Proobe- For the zero-shot setting, we use the Duain split. For the few-shot setting, we use subset of the OOD splits of each dataset.
Validation split Pvalid- In all settings, we use the original held-out validation whenever available in the datasets (Waterbirds, CelebA, MultiCelebA, ImageNet). When unavailable (CMNIST, CIFAR- 10C, 100AR- we use random half of the test splits of the datasets.
Test split Diest In all settings, we use the original test set. When half of it was used for validation due to a lack of a validation split, then we use the other half for evaluating the models.
Evaluation metries. We use the accuracy on the test set as the main evaluation metric. For datasets with subpopulation shifts, we use the worst-group accuracy (WGA), defined as the minimal accuracy over different sub- anb-populations of the dataset (Sagawa et al. 2020a).
DNN model usage and selection. We exclusively use publicly available pre-trained model weights trained on specific dataset relevant to our experiments. Importantly, we only use frozen representa- tions from these networks and do not fine-tune any parameters of the DNNs. We primarily use ViTs and ResNets in study due to their differing inductive biases. The availability of pre- trained model weights varied, and we aimed to include the most popular and nigh-performing models within each distribution shift.
5

---

Published as a conference paper at ICLR 2025
4.2 RESULTS UNDER THE FEW-SHOT SETTING
We evaluate model performance when a few labeled OOD samples are available for ILC or last-layer retraining. Our goal is challenge the assumption that the penultimate layer contains sufficient information for OOD generalization (lzmailov et al. [2022] Kirichenko et al.] [023] Rosenfeld et al. 2022) and to inspect the common practice of probing the last layer for this purpose (Zhai et al. 2020). To do so, we compare the effectiveness of intermediate representations with that of the penultimate layer.
4.2.1
INFORMATION CONTENT FOR OOD GENERALIZATION AT LAST VERSUS INTERMEDIATE LAYERS
We measure the information content in the last-layer representation versus the intermediate layers by evaluating their accuracy on OOD tasks. To quantify this, assume large number of OOD samples, meaning that the entire validation set, as defined in $4.1 is used for D Prote ~ POOD-
95.9%
ResNet18
VIT
88.1.69.44
83.1%
Accuracy
80% %
74.7.
Best layer
69.95 09.9%
%0%
44.0g 45.6%,
47.4u47.2%
40%
CMNIST
CIFAR-10C
CIFAR-100C
MultiCelebA
CMNIST
CIFAR-10C
CIFAR-100C
Dataset
Dataset
Figure 3: Information content for OOD generalization in last layer vs intermediate layers. "Last layer" refers to the OOD accuracy of the last-layer retraining approach (ILC.-1)): "Best layer" refers to the maximal OOD accuracy among the intermediate layer classifiers (ILC) (ILCC.). For MultiCelehA, we report the worst- group accuracy (WGA).
Fig. shows the OOD accuracies of the last-layer retraining approach and the best performance from the ILC across different layers. Out of the six datasets considered, observe general increase in information content when intermediate layers are utilized for OOD generalization instead of the last layers. For ResNets, the performance increments are (+16.8, +6.3, + 1.6,, +6.3) percentage points on (CMNIST, CIFAR-10, 10, CIFAR-100C, MultiCelebA). ViT has seen smaller or slightly negative increments of (+1.1, +1.3, -0.2) percentage points on (CMNIST, CIFAR-IOC. CIFAR- 100C). This point is further supported by experiments with non-linear probes (Appendix C.2 and an analysis controlling for feature dimensionality (Appendix C.3), both yielding similar findings.
We conclude that abundant information exists for OOD generalization in the intermediate layers of a DNN. The current practice of utilizing only the last layer representations may neglect the hidden information sources in the earlier layers.
4.2.2 OOD DATA EFFICIENCY FOR LAST VERSUS INTERMEDIATE LAYERS
The previous experiment measures the maximal information content at different layers with abundant OOD data train the probe; here, we consider the data efficiency for OOD generalization different layers. In practice, it is crucial that a good OOD generalization is achieved with restricted amount of OOD data. In this experiment, we control the amount of probe training set Pprobe ~ POOD of OOD samples with the parameter π E (0. 1) controlling the fraction of OOD data used, compared to the setting in (orresspnnn
1.0). While achieving the best performance is not the primary goal, we also benchmark II.C's results against other methods in Appendix A.5.2
We illustrate the ILC and last-layer retraining performances on datasets in Fig. For subpopulation shifts (Waterbirds, CelebA, MultiCelebAA,, we find that training with a smaller amount (x S 0.03) of OOD data leads to a greater empirical advantage of ILCs compared to last-layer retraining with (+5.7,+3.44 +27.0) percentage points. For the other shifts (CMNIST, CIFAR-10C, CIFAR-100C), we observe a consistent benefit of ILC compared to the last- retraining. For example, at π = 0.25, ILC boosts the performance by (+12, +3.6, +1.0) percentage points. ViTs exhibit similar pattern where ILCs perform better under little OOD data, but the difference is less pronounced (Appendix, Fig. A.5.1

---

Published as a conference paper at ICLR 2025
Waterbirds
CelebA
MultiCelebA
93.6%
90.7%
79.6%
90%
WGA
90%
85%
50%
80%
3%
37%
75%
100%
27%
54%
100%
1%
36%
100%
CMNIST
CIFAR-10C
CIFAR-100G
Accuracy
90%
95.9%
75%
Best layer ILC
Last n.C.
76.2%
45.6%
70%
30%
70%
65%
5%
25%
50%
100%
1%
20%
100%
4%
20%
40%
100%
of data
data
Figure Accuracies ILCs and last layer retraining under varying number of OOD samples ResNets. Performance of best ILCs and last-layer retraining on subpopulation shifts (first row) and the remaining shifts (second row) using CNN models. We used ResNet50 for Waterbirds and CelebA, and ResNet18 for the remaining datasets.
Takeaway from $4.2) Last layer representations often sub-optimal for OOD generalization; intermediate layers offer better candidates. The effect is more pronounced when only a small fraction of OOD samples are available to train the linear probe top.
4.3 RESULTS UNDER THE ZERO-SHOT SETTING
We now explore a scenario where no OOD samples are available for training the linear probes, but only ID samples are probe PID. This scenario is intriguing both conceptually and practically. Conceptually, it challenges us to extract features that are effective for OOD generalization using only ID data. This requires leveraging the structure of ID data to identify characteristics that may generalize well to unseen OOD) cases. Practically, the assumption that OOD data is unavailable greatly broadens potential application scenarios. We follow the setup outlined in $3.1]
We stress that this zero-shot setting was not considered in previous last-layer retraining methods (Izmailov et al. 2022 LaBonte et al. 2023), which involved re-training the last layer on OOD data.
4.3.1 SUBPOPULATION SHIFTS
We illustrate results on subpopulation shifts in Fig. 5 We consider three variations of models: (1) the pre-trained frozen DNN (Base), (2) last- layer re-training with the ID data (Last layer), and (3) the best ILC after training on ID (Best layer). They can be compared as they solve the same task. We also compare these zero-shot results to performant baselines in appendix[A.6.
100%
87.1
82.0
69.3
56.0
WGA
50%
45.0
17.620.8
0%
Waterbirds
CelebA Last layer ILG
Multi-CelebA Best layer ILC,
Base
In all three datasets, we observe the relation- ship: Base<Last layer<Bestt layer. First of all, the relationship Last layer is intriguing because just by re-training the last layer of pre- trained model on the same training set improves the OOD generalization performance. This remark out of the paper's scope, but exploring this phenomenon may result in novel insights. Wide gaps are seen for Last layer<Best layer, particularly for Waterbirds (79.4% -87.1%)) CelebA (56.0%-+82.0% and MultiCelebA (20.8%-+46.0%
Figure 5: WGA OOD data for ID-trained CNNs. the explanations Base, Last layer, and Best layer, refer the text. ResNet50: Waterbirds, CelebA ResNet18: MultiCelebA..
We conclude that under subpopulation shifts, intermediate layers can be effectively leveraged using only ID data. In contrast, last-layer retraining may fall short, likely because the features in the final layer are already over-specialized to the training distribution, possibly due to the memorization of minority datapoints (Sagawa al. 2020a).. Our results suggest that intermediate layers may not suffer from this issue to the same extent.

---

Published as a conference paper at ICLR 2025
CIFAR-10C
CIFAR-100C
71x 69
69h 65
61g 56%,
67% 63g
65% 63
67% 64%
67 63%
Accuracy
43v 42%
35% 32
60%
30%
Best layer ILC
Last layer
0%
ILC,
VIT
MLP-Mixer
GNet
R-18
R-50
DN-121 DN-169
ViT
R-18
Figure 6: CIFAR-IOC and CIFAR-10OC results for zero-shot ILC last-layer retraining. We compare the best layer and last layer performances multiple model architectures. Since there are 19 noise types, plot the distribution accuracies with violin plots showing the kernel density estimation outputs. We link results for individual noise types across best layer and last layer results with solid lines.
4.3.2 INPUT PERTURBATION SHIFTS
We show the zero-shot performance ILC and last-layer retraining on OOD shifts with input perturbations. Results for CIFAR-I0C and CIFAR-100C are provided Fig. 6 We report the performance of multiple models with publicly available weights, including ResNet-18 and ViT. Detailed descriptions of these models can be found in Appendix A.1
On CIFAR-10C all models show +2%p to +5%p improvements in mean accuracies when the best layer is used, instead of the last layer. ViT performs best under the ILC framework, reaching an average accuracy of 71%. For 10FAR-100C, the accuracy gains are +3%p for ResNet-18 18 and +1 +1%p for the ViT model. These results consistently demonstrate the benefits of utilizing intermediate-layer representations for improving OOD generalization across different model architectures.
4.3.3 STYLE SHIFTS
We present the zero-shot relative average accuracies between ILC and last-layer retraining in Fig. For cue-conflict and silhouette style shifts, the average accuracy boosts for ILC are +2 percentage points. For Imagenet-A and ImageNet-R, the average relative accuracies are +0.91 and +0.34 percentage points, respectively. The classes silhouette and cue-conflict datasets are more abstract and require the model to rely other cues, such as shape, which could explain the higher relative improvements in those cases.
Relative acturacy [%hp]
Acc(llCr-) AActtlC,, Cc
R-18
R-50
R-101
VIT-B
ConvNeXt-B
Swin-B
DN-121
DN-169
5.9
6%
4.4
3%
1.4
0%
0.20.8
0.344
11122a
0.4 0.5
0.3
Cue Conflict
Silhouette
IN-A
IN-R
Figure 7: Performance boost for intermediate layers ImageNet variants for zero-shot setting, We show the performance of the best intermediate layer relative to the last layer.
Takeaway from 4.3 OOD performance can be improved using ILCss only using ID data as long held-out dataset available for model selection.
ANALYSIS
In this section, we analyze the contributing factors behind the improved generalization performance of the ILCs shown in the previous section. We analyze the impact of depth and corresponding distances between embeddings of in-distribution and out-of-distribution samples.
8

---

Published as a conference paper at ICLR 2025
5.1 IMPACT OF DEPTH
We inspect the contribution of depth to the generalizability of linear probes trained on intermediate representations, We consider both few-shot and zero-shot scenarios, according to the availability of OOD data ) For the few-shot case, we consider the oracle model selection case due to training ILCs on the whole Dvalid. We report the performances of ILCs trained on all layers in Fig.
We observe two key patterns:
(I) The pen-penultimate layer consistently outperforms the penultimate layer. Across all datasets, the pen-penultimate layer (layer 7) achieves higher OOD accuracy than the penultimate layer (layer 8) in both few-shot and zero-shot scenarios. For instance, in the Waterbirds dataset, the 7th layer ppen-penultimate attains a worst-group accuracy (WGA) of 94.0% in the few-shot scenario and 86.7% % the zero-shot scenario, compared to the 8th layer's (penultimate) 93.9% and 79.4%, respectively. This indicates that representations from the pen-penultimate layer generalize better under distribution shifts than those from the penultimate layer.
(2) The best-performing layer varies across datasets. The optimal layer for ILCs depends on the distribution shift and dataset. In CelebA and Multi-CelebA,, earlier layers outperform the penultimate layer. For example. the 5th layer yields 80.6% WGA in the zero-shot scenario for CelebA, and the 6th layer achieves 79.5% WGA in the few-shot scenario for Multi-CelebA. We observe similar patterns of depth impact other datasets like CIFAR-10 and CIFAR-100: detailed results for these datasets using Re models are provided in the Appendix (see Fig. 17
Waterbirds
CelebA
Multi-CelebA
100%
29.8
WGA
50%
Zero-shot
0%
3 4 5 6 7 8
5
8
2
Layer
Layer
Layer
Figure 8: OOD accuracies across layers. We consider subpopulation shifts (Waterbirds, CelebA, and Aulti-CelehA) cases when OOD data available ('Few-shot') and when it is not ("Zero-shot'..
Takeaway from $5.1 When training ILCs a single layer, classifiers built on intermediate representations paess--patcculalyyy the pen-penultimate layer-often outperform those built on the penultimate layer.
5.2 FEATURE SENSITIVITY INTERMEDIATE AND PENULTIMATE LAYERS
We observed that in 1 the few-shot setting, ILCs consistently outperform last-layer retraining, which relies on features from the penultimate layer. This observation suggests that the information content in penultimate layers is smaller compared to the feature in the intermediate layers.
We hypothesize that the increased performance is not only because these features are inherently more informative about the factors of variation within the data but also because they exhibit greater invariance distribution shifts compared to those in the penultimate layer. According to this hypothesis, for example, if an image is perturbed by small amount of noise, the features in the penultimate layer may change significantly, while the features the intermediate layers may remain more stable. This would explain ILCs' better performance in the zero-shot setting ($4.3): classifiers built on top of ID features would transfer better the features derived from OOD data were similar to those derived from ID data.
In this section we test this hypothesis. We focus our analysis on subpopulation shifts here; similar results hold for input-level shifts (Appendix B.2.
Intermediate layers are less sensitive to distribution shifts. We test the hypothesis that intermediate layers are less sensitive to distribution shifts. We define intuitive global metric to compare how far test data points deviate from the training data at each layer. This is done by comparing the average distances between points in the training set and those in the test set. The ratio of these distances gives a sense of how much a layer "notices" the shift. This helps us understand which layers remain stable under distribution shifts and which are more sensitive.

---

Published as a conference paper at ICLR 2025
Layer
Layre
Waterbirds
CelebA
MuttCCalebA
NO0RR Sttooiky
PNC 2
seinttiiity
Minority
0.30
dios
Matti 0.20
0.32
0.
Majority
0.03
0.04
0.15
PC2
0.00
2.00
SYTEA
0.01
0.00
PC
FC1
PC
Layer
Layer
Layer
Toxin
Text
Figure 9: Sensitivity to subpopulation shifts across layers. The "Minority" refers to the least frequent group in the datasct (dark blue). The "Majority" refers to the most frequent group (light blue). Groups in between are colored in shades blue.
Figure 10: ID and OOD separation in intermedi- ate layers (ResNet18 on MultiCelebA). PCA projec- tions for minority (top) and majority (bottom) groups show increasing test-train separation in earlier layers, with layer 8 showing the largest separation for one minority group.
Concretely, we measure the mean pairwise distance between the training points and testing points belonging to the same group g (e.g. a group of blond males in CelebA), and normalize this quantity by the mean pairwise distance between the probe points within that group. The datasets 'D Drohe, a Duss refer to the training and testing sets from group 9, respectively. We define the mean pairwise distance di (Di, D2) and the sensitivity score sens; for layer as:
and
(Lst((Di,Da)
1
Σ 42
-
- rn(()))
sensi == di sistt sist (Deotte Dusty) dist: (Drrhe, Drrcte
D1 D2
where r (D( (1) and r; (D( denote the representations of the individual samples i and from the datasets at layer I; The sensitivity score sens; of 0 indicates that the test points are as far from the training points as the training points are from each other. A high sensitivity score indicates that, on average, the test representations are more separated from the training points, reflecting greater sensitivity to distribution shifts.
We illustrate the sensitivity scores between ID and OOD data for the subpopulation shifts in Fig. 9 We observe two key patterns:
(1) Minority groups are more sensitive to distribution shifts. Features extracted from OOD data (i.e., minority groups) tend to have higher sensitivity scores compared to those from majority groups, particularly at deeper layers. This indicates that underrepresented samples from minority groups are mapped further away from the IID training samples in feature space compared majority groups.
(2) Intermediate layers are less sensitive to shifts compared to the penultimate layer for mi- nority groups. Across both CelebA and MultiCelebA, observe that intermediate layers exhibit significantly lower sensitivity scores, often collapsing to single low value, while the penultimate layer remains more sensitive to subpopulation shifts. This suggests that intermediate layers are more invariant to distribution shifts, particularly for underrepresented groups, while the penultimate layer tends to overemphasize these shifts.
In Fig. 10 we illustrate the separation between ID and OOD datapoints qualitatively. The projections of minority and majority group datapoints show that for the minority group, highest separation between testing and training points occurs at the penultimate layer, with earlier layers clustering points closer together. The majority group shows less separation across layers.
Takeaway from $5.2 The reason ILCs outperform last-layer retraining on OOD generalization tasks may be due to the robustness of intermediate layers to distribution shifts.
CONCLUSION
While prior work has shown that last-layer retraining can be sufficient for certain OOD generaliza- tion tasks, our results indicate that this approach can sometimes fall short. We demonstrate that intermediate-layee representations are less sensitive to distribution shifts than the penultimate layer. By introducing ILCs, which capitalize on this reduced sensitivity to distribution shifts, we provide stronger baseline for zero-shot and few-shot learning. These findings suggest need to reassess the focus on penultimate- layer features and leverage the stability of intermediate layers for more effective generalization in complex, real-world scenarios.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
We thank the anonymous reviewers for their valuable feedback, and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Arnas Uselis. This work was supported by the Tübingen AI Center. We also acknowledge Alexander Rubinstein and Ankit Sonthalia for their helpful insights.
REFERENCES
Thomas Adler, Johannes Brandstetter, Michael Widrich, Andreas Mayr, David Kreil, Michael Kopp. Günter Klambauer, and Sepp Hochreiter. Cross-Doman Few-Shot Learning by Representation Fusion. Number arXiv:2010.06498. arXiv, February 2021.
Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. Number 111v:1610.01644.. arXiv, November 2018. doi: -485500arXxv.1610.0111.
2 3.2 A.4 B.I.1
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data represen- tations in deep neural networks. Number 19X.::1905.12700 arXiv, October 2019, doi: 10.48550/arXiv. 1905.
12784. 2
Martin Arjovsky, Léon Bottou. Ishaan Guirajani. and David Lopez-Paz Invariant Risk Minimization. Number arXiv:1907. 02293,, March 2020. doi: (0.48550/arXiv.19777 1907.02877.......
A.4
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jacgul Choo, and Scong Joon Oh. Learning De-biased Representations with Biased Representations. Number arXiv:1910.02806. arXiv, June 2020. doi:
114485500rXXiv.1110000238 2
Sara Beery, Grant Hor,, and Pictro Perona. Recognition in Terra Incognita. Number 1ww.vv::110.0997771 arXiv, July 2018.
Remi Cadene, Corentin Dancette, Hedi Ben-younes, Matthieu Cord, and Devi Parikh. RUBi: Reducing Unimodal Biases in Visual Question Answering. Number 1exx:xv:196666 10169. arXiv, March 2020. [
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. arXiv:2010. 11929 [cs]. June
2021. 3.2 A.7
Luciano Dyballa, Evan Gerritz, and Steven W Zucker. A separability-hasse approach to quantifying generaliza- tion: which layer is best? arXiv preprint arXiv:2205.011 524, 2024. 2
Utku Evci, Vincent Dumoulin, Hugo Larochelle, and Michael C. Mozer. HeadZToe: Utilizing Intermediate Representations for Better Transfer Learning. Number arXiv:2201.033: arXiv, July 2022. doi: 10.48550/ arXiv.2201.033. 003529.
2
Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:21 12.12, 2021. CT
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaclis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A. Wichmann. Shorteut Learning Deep Neural Networks. Nature Machine Intelligence, 2(11): 665-673, November 2020. ISSN 2522-5839, doi: 10. (038//42256-000000000
1
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. Number arXiv: 11t.::11111122111 11.133.1. arxiv, November 2022. doi: 10.48550/arXiv. 1811..2231...
Evan Gerritz, Luciano Dyballa, and Steven Zucker. Zero-shot generalization across architectures for visual classification, arXiv preprint Ail.:2402.14099 2024. p
Ishaan Gulrajani David Lopez-Paz. Search of Lost Domain Generalization. Number arXiv:2007001334.
arXiv, July 2020. 3.1
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. Number arXiv:15 512.03338...... arXiv, December 2015. doi: 10.48550/arXiv.1555 151..e.....1111....... 3.2
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. Number arXiv:1903. 12261. arXiv, March 2019.
11

---

Published as a conference paper at ICLR 2025
Dan Hendrycks. Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The Many Faces of Generalization. Number 2tti::2000.112444. arXiv,
Robustness: A Critical Analysis Out-of-Distriben July 2021a. doi: 00448500arxiv.200..... 16241.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural Adversarial Examples.
Number arXiv:1907. Net::1907.077444 07174, arXiv, March 2021b. 2 A.7
Katherine Hermann and Andrew Lampinen. What shapes feature representations? Exploring datasets, archi-
tectures, and training. In Advances in Neural Information Processing Systems, volume 33, pp. 9995-999000 Curran Associates, Inc., 2020. 2
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4705 2017. A.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q Weinberger. Densely Connected Convolutional Networks. Number arXiv: 608.06993. arXiv, January 2018. doi: 10.48550/arXiv. 1608 06993.6993.9 A.7
Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing Adversarial Example Transferahility with an Intermediate Level Attack. Number arXiv: 11KI::1197..0000 10823. arXiv, February 2020. doi: (0-48500axXX..100..1083
Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew G Wilson. On feature learning in the presence of spurious correlations. Advances in Neural Information Processing Systems, 35:38516-38855 2022. 2
3.1
3.2
A.5.2
Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the Generalization Gap in Deep Networks with Margin Distributions. Number 10Xv:18100000111 arXiv, June 2019, doi: 10,48550/arXiw. 1810.00113.
Bingyi Kang, Saining Xic, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling Representation and Classifier for Long- -ailed Recognition, Number arXiv:1910.09217.1 arXiv, February 2020. doi: 00.44850/arxiw.1100.22
3.2
Nayeong Kim, Juwon Kang, Sungsoo Ahn, Jungseul Ok, and Suha Kwak. Removing Multiple Shorteuts through the Lens of Multi-task Learning. October 2023. 2 A.5
A.9
Diederik Kingma and Jimmy Ba. Adam: Method for Stochastic Optimization. Number arXiv:1412.699 1412.698.. arXiv, January 2017, doi: 10.48550/arXiv.. 1412.698...8 A.2.2
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations. Number arXiv:2204002937.c arXiv, June 2023. doi: 10.48550/arXiv. 2204.02937.
4.2 A.2.1 A.2.2
A.6
A.9
Alex Krizhevsky, Leaming Multiple Layers of Features from Tiny Images. 2000. A.2.1
Tyler LaBonte, Vidya Muthukumar, and Abhishek Kumar. Towards Last-layer Retraining for Group Robustness with Fewer Annotations. Number arXiv:2309.0853 arXiv, September 2023. doi: 10.485500arXiv.2300 0353444434.........
Lecun, L. Bottou, Bengio, and Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, (6(11):2278-233 November 1998. ISSN 1558-2256. doi: 10.1 109//.777777 A.4
Yoonho Lee, Annie S. Chen, Fahim Tajwar, Ananya Kumar, Huaxiu Yao, Perey Liang, and Chelsea Finn. Surgical Fine- Pine-Tuning Improves Adaptation to Distribution Shifts. Number arXiv:2210.1 1466. arXiv, June 2023. doi: 10.48550/arXiw.22100. 11..2221....6..111 2
Xiao Li. Sheng Liu, Jinxin Zhou, Xinyu Lu,, Carlos Fernandez- Granda, Zhihui Zhu, Qing Qu. Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint 22X 12206, 2022.
C.I
Zhiheng Li. Ivan Evtimov, Albert Gordo, Caner Hazirbas, Tal Hassner, Cristian Canton Ferrer, Chenliang Xu, and Mark Ibrahim. A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Others. Number Sitii:2212..0455 arXiv, March 2023, doi: 10.48550/arXiv.222 htt..2222..44.......
Evan Zheran Liu, Behzad Haghgoo, Annie S. Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just Train Twice: Improving Group Robustness without Training Group Information. Number Xiv:22077.09044. 09044. arXiv, September 2021a A.6
12

---

Published as a conference paper at ICLR 2025
Ze Liu, Yutong Lin. Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. Number 1IXx:2103311430 arXiv,
August 2021b. doi: 10.48550/r2xw.21033. 14030, [A.10]
Zhuang Liu. Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. ConvNet for the 2020s. Number arXiv:2201 2.05552.0..0.55..... arXiv, March 2022. doi: 10.485500/rXiw.2201 .03545. AA.10
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep Learning Face Attributes the Wild. Number aXiv:1411.77666 arXiv, September 2015, doi: 100.550/aXii.1411..77000 2
Torch Vision maintainers and contributors. TorchVision PyTorch's computer vision library. November 2016.
Wojciech Masarczyk, Mateusz Ostaszewski, Ehsan Imani, Razvan Pascanu, Piotr Milos, and Tomasz Trzcinski. The tunnel effect: Building data representations in deep neural networks. Advances in Neural Information Processing Systems, 36:76772 76805, 2023. 2
Aditya Krishna Menon, Ankit Singh Rawat, and Sanjiv Kumar. Overparamottriisation and worst-case generalisa- tion: Friend or foe? In International Conference on Learning Representations, October 2020. P
Vardan Papyan, X. Y. Han, and David L.. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, (24-)))-222222246666 October 2020. doi: 10.
C.
Juhyeon Park, Seokhyeon Jeong, and Taesup Moon. TLDR: Text Based Last-layer Retraining for Debiasing Image Classifiers. Number arXiv:231 188291 arXiv, November 2023. 2
Mohammad Pezeshki, Gékou-Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient Starvation: A Learning Proclivity in Neural Networks. Number arXiv:201 1.09668. arXiv,
November 2021. 2
Akshay Rangamani, Marius Lindegaard, Tomer Galanti, and Tomaso A. Poggio. Feature learning in deep classifiers through Intermediate Neural Collapse. In Proceedings of the 40th International Conference on Machine Learning, pp. 28729-287455 28745. PMLR, July 2023.
Stefano Recanatesi, Matthew Farrell, Madhu Advani, Timothy Moore, Guillaume Lajoie, Eric Shea- thea-Brown. Dimensionality compression and expansion in Deep Neural Networks. Number Xiv: 10w.004044443... arXiv, October 2019. doi: 10.48550/arriv.. 10..0XX9111999333 1996.0044.. 2
Benjamin Recht, Rebecca Roclofs, Ludwig Schmidt, and Vaishaal Shankar. Do ImageNet Classifiers Generalize to ImageNet? Number 19Xiv:1902.110 10811. 1. arXiv, June 2019. doi: 10.48550/arXiv.1900 1..
Amir Rosenfeld, Richard Zemel, and John K. Tsotsos. The Elephant in the Room. Number arXiv:1808.03300 1808.0330.
arXiv, August 2018. 0
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain Anjjuttd Regression or: ERM May Already Learn Features Sufficient for of Distribution Generalization. Number arXiv:2202.06855 arXiv, October 2022. doi: 00.48550Marii.22222.6666
Yangjun Ruan, Yann Dubois, and Chris J. Maddison. Optimal Representations for Covariate Shift. Number arXiv:2201 .00057.00 arXiv, March 2022.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh. Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. Number arXiv:1409.0s. 1409.0575. arXiv, January 2015. doi: 14.48550/rXiiv14400.5555
A.2.1
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization. Number arXiv:191 1, 1,08731. arXiv, April 2020a. doi: 10.48550/arXiv. 191 1.111..0000 1.08311.
2 3.1
4.3.1
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An Investigation of Why Overparam-
eterization Exacerbates Spurious Correlations. Number arXiv:2005.04345 arXiv, August 2020b. doi: 10. 10.485550axXiv.200..4444
Luca Scimeca, Scong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which Shortcut Cues Will DNNs Choose? A Study from the Parameter-Space Perspective. Number 1=Xiv:2110.03000 10.03095. arXiv, February 2022. doi: (0.48550/arXiv2211 10..0399.
2
13

---

Published as a conference paper at ICLR 2025
Yuge Shi, Imant Daunhawer, Julia E. Vogt. Philip H. S. Tom, and Amartya Sanyal. How Robust is Unsupervised Representation Learning to Distribution Shift? Number arXiv:2206.080 arXiv, December 2022. [
Sahil Singla and Soheil Feizi. Salient ImageNet: How to discover spurious features in Deep Learning? In
International Conference on Learning Representations, October 2021. !
Ankit Sonthalia, Alexander Rubinstein, Ehsan Abbasnejad, and Seong Joon Oh. Do Deep Neural Network Solutions Form a Star Domain? Number arXiv:2403.07968 arXiv, March 2024, doi: 10.48550/a2xiv2240...
07968. A
Christian Szegedy, Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going Deeper with Convolutions. Number 1rXivv:10994485 arXiv, September 2014. A.7
Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt, Mea- suring Robustness to Natural Distribution Shifts in Image Classification. Number arXiv:2007.0064: arXiv, September 2020. doi: (0-48550/rXiv.2007.0666
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-Mixer: An all-MLP Architecture for Vision. Number arXiv:2105.0160) arXiv, June 2021. doi: 10.48550/arXi..2105.
01601. A.7
Guillermo Valle-Pérez, Chico Camargo, and Ard A. Louis. Deep learning generalizes because the parameter- function map is biased towards simple functions. May 2018.
Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, Toannis Mitliagkas, Aaron Courville, David Lopez-Paz, and Yoshua Bengio. Manifold Mixup: Better Representations by Interpolating Hidden States. Number arXiv:18006 100.00536.. arXiv, May 2019. doi: 10.48550/arXiv. (0t.://araxx1110002222222 180.00233.
Yuzhe Yang. Haoran Zhang, Dina Katabi, and Marzyeh Ghassemi. Change is Hard: A Closer Look at Subpopulation Shift. Number arXiv:2302. 12254. arXiv, August 2023. doi: 10.48550/axxiv.3302.2222 P
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. How transferable are features deep neural networks? Advances in neural information processing systems, 27, 2014. 2
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features. Number M.XXXv:910500880 arXiv, August 2019. doi: 10.48550/arXiv11990 1905.04899
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A Large-scale Study of Representation Learning with Visual Task Adaptation Benchmark. Number arXiv:1910.04867, arXiv, February 2020. doi:
Hongyi Zhang, Moustapha Cisse, Yann Dauphin, David Lopez-Paz. Mixup: Beyond Empirical Risk Minimization. Number arXiv: arXiv, April 2018. doi: 11483500arXiix..10.0444
Michael Zhang, Nimit S. Sohoni, Hongyang R. Zhang, Chelsea Finn, and Christopher Ré. Correct-N-Contrst Contrastive Approach for Improving Robustness to Spurious Correlations. Number 2rXiv:2203.01517. arXiv, March 2022. doi: 10. 21.5500/rXii.2220011333 A.6
14

---

Published as a conference paper at ICLR 2025
A DETAILS ON MAIN RESULTS
A.1 COMPOSITION OF USED MODELS
We detail the layers of the main models considered throughout this study. We use ResNet variants from the Torch Vision library (maintainers & contributors, 2016). The models from the ResNet architecture family can be represented as a composition blocks, e.g., for the ResNet-50 model, we have:
ResNet 50 := Block o Block Block".
More specifically, can describe the structure of ResNet-18 and ResNet-50 as follows:
ResNet-18 == Conv(64, > (BatchNorm + ReLU) + MaxPool (3,2) - Res 2) (es8lock(128,2)
ResBlock(256,2 ~ ResBlock(512,2) + Adaptive
ResNet-50 7onv (BatchNorm + MaxPool
( + ResBlock 4)
ResBlock 6) > ResB1ock(512,35 AdaptiveAvgPool,
where:
Conv(filters, kernel size, stride): convolutional layer,
BatchNorm: batch normalization layer,
LU: rectified linear unit,
MaxPool(pool size, stride): max pooling layer,
ResBlock(filee) number of blocks): residual block, consisting of a bottleneck structure with three convolutional layers (Ix1, 3x3, Ixl), each followed by batch normalization layer, and ReLU activations after the first two convolutions, plus skip connection,
Adaptive PeAvPPoll: adaptive average pooling layer.
Note that our description of the models excludes the last layer corresponding to the classification head.
A.1.1 CIFAR-SPECIPE RESNET MODELS
For CIFAR-10 10 and 100, the ResNet architecture is modified to account for the smaller input size (32x32 instead of 224x224). Key differences are:
1. Removal of the initial 7x7 convolution and 3x3 pooling,
2. Reduced stride from 2 to 1.
A.1.2 VIT FOR CIFAR-10
The architecture of the Vision Transformer (ViT) for CIFAR- is follows:
ViT CIFAR- = EmbeddingLayer (192, 384) + EncoderLayer +
EncoderLayer - incoderLayer" + EncoderLayer
EncoderLayer' + EncoderLayer* + EncoderLayer +
(2)
EncoderLayer' + EncoderLayer* + EncoderLayer
CLSExtraction - Norm (384)
Where:
EmbeddingLayer fn__utesses, out_featured Linear layer with input size 192 and output size 384.
15

---

Published as a conference paper at ICLR 2025
EncoderLayez: Composed of:
LayerNorm (384)
Multi-head Self-Attention (384 dimensions)
MLP (384, 384) with CELU activation
eusExtraction Extracts the [CLS] token for classification.
Norm (384): Layer normalization with feature size 384.
FC (384, 10): Fully connected layer mapping 384 features to 10 output classes (CIFAR-
10).
A.1.3 VIT FOR CMNIST
The architecttre of the Vision Transformer (ViT) for CMNIST is as follows:
ViT CMNIST == EnbeddingLayer((47, 384) - EncoderLayer" + EncoderLayer's +
EncoderLayer > EncoderTaayr" + EncoderTayer"
CLsExtraction Norm(384)
A.2 EXPERIMENTAL SETTING
A.2.1 DATASET SPLITS IN ZERO- AND FHW-SHOT SETTINGS
Here outline the construction of datasets and the implementation of zero- and few-shot settings.
Few-shot setting. In the few-shot case, we assume the availability of OOD samples for either training ILCs or performing last-layer retraining, D Prrobe. as well as validation set Dvalid of OOD samples for selecting the best layer.
We consider two settings: setting for deriving hound performance of last-layer retraining and ILCCs, and a setting where OOD samples for training ILC.s is limited. In the first setting we assume
oracle model selection process, that is, valid D test.
CMNIST dataset has an associated OOD testing set. We uniformly sample half of it and assign to to Drohhe and the remaining half to Diest- CIFAR-1OC and CIFAR-100C have 19 noise types associated with them. We carry out our experiments for each distribution shift separately, essentially treating each noise level as its own OOD dataset. For each of the datasets, we split them into two equally sized subsets: Pprobe and Dres. We train ILCs and perform last-layer retraining on Pprobe. and test them on Diea- For each of Waterbirds, CelebA, and MultiCelehA, there is an associated held-out set Dheld ou and testing set Dies. The held-out set takes form of D out = Uf Di Dieetd-ou with samples that have associated group labels. Following (Kirichenko et al. (2023), we take this held-out set and balance it with respect to the number of samples within each group. We achieve this for each dataset by taking the number of samples the smallest group and subsampling the other groups to match this number using uniform sampling. This results in D Prooee + u. Di Dalaancdd with DDiatancca = balanncca for j & G, and D Dialanced S Dheld-out that we use for training the ILCs. For testing, we use associated test sets that come with these benchmarks Diest, taking form of Diest = U Diss
In the second setting we consider scenario where IICCs are trained on D Drosee where only a fraction of data E (0, 1) from Protbe are assigned to D Pirone's 1 proportion of the data from the original Dprshe is used for validation. That is, we divide Drroh D probe U Valid with [Ppmhhel
| prophe J--
and use DT Pr sample a proportion of π datapoints from D probe and assign them to D probe- and assign the remaining points to D
D Valid for training and validating the ILCs, respectively. In words, uniformly
Zero-shot setting. In the zero-shot case, we assume that no OOD data is available for training ILCs or performing last-layer retraining. Instead, we assume that only the ID data on which the DNN model is trained is available for training the ILCs, i.e. D Proibe < D For CMNIST, this amounts to using the biased data the DNN model was trained on. For 1OC and CIFAR-100C.
'This Dheid-an our is usually referred to validation set. To avoid confusion with validation set used for model selection, we refer this set held-out set.
16

---

Published as a conference paper at ICLR 2025
Proobe corresponds the CIFAR=10 and C10 datasets (Krzhevssky 2000
respectively. For
Waterbirds, CelebA, and MultiCelebA, it amounts to the original group- -oou--mbblannedd dataset used for training the DNN, where group labels are hidden from ILCss For ImageNet variants, this amounts to either training the II.Cs or performing last-layer retraining on the original ImageNet- K dataset (Russakovsky al. 2015).
We only assume an OOD set for performing validation. This validation dataset consist of same samples used for training the probes for the few-shot case: for CMNIST it is half of the OOD testing set, for CIFAR it is half of each of the OOD datasets, and for Waterbirds, CelebA, and MultiCelebA, it is the original held-out dataset. Due to the size of OOD sets of ImageNet variants being relatively small, we perform oracle model selection on it.
A.2.2 HYPERPARAMETER SEARCII
We perform minimal hyperparameter search over the learning rate and & regularization strength (similarly to (Kirichenko et al., 2023)) when training ILCs, For the zero-shot case, we tune the learn-
ing rate l7 and & regularization strength according to (17, E1) E Hzero-shor = {10 - 1003, 110001000000000001000111000 10020-- 2 x
{0, 11 10 10-21.1--111 J. For the few-shot case, we use (n, E Heewsshox
{10 10-34,0033, 10 x
{0, 10 10 10 1002),, since we found higher regularization rates to help last-layer retraining.
For all experiments, we use the Adam optimizer (Kingma & Ba (017), training ILCs and performing last-layer retraining for 100 epochs. We repeat each experiment at least three times, varying the DNN model's seed, the initialization of the ILC.s, and the data splits, depending on the specific setting. Depending on availability, we use either Nvidia 2080 or A100 GPUs.
A.3 TRAINING AND INFERENCE WITH ILCs
Once the best intermediate layer is determined, inference can be simplified by using only the cor-
Algorithm 2 Inference with ILC for (-th layer
responding ILC, as described in Algorithm 4 In- stead of propagating through the entire network, inference at layer involves computing the inter- mediate representation at the 1-th layer and pre-
1: Input: Input sample pre-trained network f. weights W,, biases b intermediate layer I
2: Compute intermediate representation (x)
(fi o..... o
fx)
3: Compute logits ý1 = Wir((x) + b;
dicting the output based on the ILC of that layer. This enables efficient computation by avoiding un- necessary forward passes through the remaining layers fit1...., fL.
4: Output: Predicted label ÿ = arg myx
A.4 ILLUSTRATIVE TOY EXAMPLE ON A CONDITIONAL SHIFT FOR INTEMEDIATE LAYERS UTILIZATION
In the following we illustrate the main ideas of this work in a self-contained manner on Colored- MNIST dataset.
We consider the Colored MNIST (Arjovsky et al. 2020 a dataset with a conditional distribution shift (where P Prain (Y) X) * Puea y X), but (X) = Pres( (X)) constructed using the MNIST (Lecun et al., 1998) dataset that artificially creates spurious correlations during training that are not useful for generalization
test time (detailed in Appendix). The training data consists of colored digit images and binary labels indicating whether the digit is 2 5 or < 4. Some of the training data is corrupted, where the label is flipped. Digit colors are the spurious cues that are more informative than the digit itself during training, At test time, color is de-correlated with the task label. The aim is to obtain a robust representation that encodes both the digit and color information, such that depending on the distribution shift, either the digit or the color can be adopted.
17

---

Published as a conference paper at ICLR 2025
ResNet-18
VIT
Few-shot
Zero-shot
Few-shot
Zero-shot
000
000 acc
0.96
0.96
0OD acc
0.90
0.8
0.950
0.90
0.87
0.84
0.94
0.84
0.925
2
1 4
2
Layer
Layer
Layer
Layer
Figure 11: Qualities intermediate representations for Colored MNIST. We show the accuracies intermediate layers of ResNet-18 18 and ViT on Colored MNIST. We consider few-shot and zero-shot settings, where the OOD samples are available and unavailable, respectively. Error bars generated over 3 random seeds. Intermediate representations offer better OOD generalization than the last.
We illustrate the performance of the model across layers when OOD data not available and when is for ResNet-18 8 and VIT (whose compositions are defined above) model with the same depth of layers in Fig. 11 we use 2,000 samples total for training the probes.
We observe that (1) when OOD data is available ('Few-shot'), the best location to train the linear probe at an intermediate layer (layer for the ResNet, and layer 7 for ViT) and relying on the last layer gives suboptimal results.
(2) When OOD data is not available ( ("Zero-hoot', noted as 'OOD'), intermediate layers are still better than the last layer; also, the same layers that were optimal when OOD data was available are still optimal when it is not (layers 6 and for ResNet and ViT, respectively).
Finally, (3) the upside of intermediate layers only for the OOD data in-distribution data is best classified at the last layer ("Zero-shot', noted as 'ID'), aligned with the findings of previous work
(Alain Bengio 2018).
A.5
ADDITIONAL FEW-SHOT RESULTS
A.5.1
FEW-SHOT RESULTS VIT
We illustrate accuracies for ViT function of fraction of OOD data used for either training the ILCs or performing last layer retraining in Fig. 12
CMNIST
CIFAR-10C
CIFAR-100C
89.4%
76.0%
90%
Accuracy
40%
85%
70%
60%
20%
5%
25%
50%
100%
1%
20%
100%
4%
20%
40%
100%
% data
data
%6 of data
Figure 12: Accuracies of ILCs and last layer retraining under varying number of OOD samples for VIT.
A.5.2 FEW-SHOT: COMPARISON OF ILCS AGAINST OTHER METHODS
We compare ILCs against last-layer retraining (zmailov et al. 2022) and an approach using multi-
object optimization (MOO) by (Kim al. 2023) in Table 3 We observe that ILCs outperform last-layer retraining and MOO across all datasets while using only a fraction of data.
Table 3: Few-shot results when probes trained using varying amount of OOD data and selected using held-out validation set of OOD data; when 100% data is used, validation performed on the data the probes are trained on. Our approach matches or exceeds SOTA for subpopulation distribution shifts, MOO results from (Kim et al. 2023
Dataset
Data Portion Used (%)
DFR (%)
MOO (%)
ILCs (%)
Waterbirds Waterbirds
75
92.2 + 0.8
N/A
93.0 + 0.8
100
92.9 + 0.5
N/A
93.0 + 0.4
CelebA
54
87.9 + 1.0
N/A
89.2 + 0.4
CelebA
100
88.6 +: 0.4
N/A
89.7 1 0.6
MultiCelebA
20
73.9 + 0.2
N/A
79.0 + 0.4
MultiCelebA
100
76.4 + 0.2
77.9 + 0.2
79.6 + 0.2
18

---

Published as a conference paper at ICLR 2025
A.6 ZERO-SHOT: COMPARISON OF ILCs AGAINST OTHER METHODS
We compare ILCs against "Just Train-Twice" (JTT) (Liu et al. 202 "Correct-N-Contrast" (CnC) (Zhang al. 2022). and DFR (Kirichenko al. 2023).
The results are illustrated in Table 4 We note that IICss outperform DFR and JTT, while only underperforming against CnC, which requires training a specialized model. In contrast, ILCs were trained on the base ERM model without any feature-extractor parameter tuning.
Table 4: Zero-shot results when probes trained using ID data and selected using held-out validation set of OOD data. Our approach competitive with complex methods.
Dataset
ERM (%)
JTT
CnC (%)
DFR (%)
ILCs (%)
Waterbirds
74.9
86.7
88.5
79.4
87.1 = 0.8
CelebA
46.9
81.1
88.8
56.0
82.0 = 1.1
A.7 CIFAR-1OC EXPERIMENTS
CIFAR-C (Hendrycks et al. 2021b consists of 16 types of noise types: (1) brightness, (2) contrast, (3) defocus, (4) elastic, (5) fog, (6) frost, (7) gaussian, (8) glass, (9) impulse, (10) jpeg. (11) motion, (12) pixelate, (13) saturate, (14) shot, (15) snow, and (16) spatter. This benchmark consists five levels of noise perturbations of the images in the validation set; we consider only the highest-intensity noise levels for each of the noise types.
Finetuning on OOD data. We illustrate each probe's performance across layers when tuning on 1024 points of OOD data for each noise type (Fig. 13) and when tuning on 128 points of OOD data for each noise type (Fig. 14].
Training on ID data results. We pre-trained models from the public repository. For the models
we train ourselves, in particular: ViT (Dosovitskiy et al.) (021) and MLP -M1 xer (Tolstikhin et al.) 202I) we used code following: ViTs for CIFAR public repository and MLP-Mixer for CIFAR public repository
We consider pre-trained ViT (Dosovitskiy et al. 2021), MLP-Mi xer TTosstikhin et al., 2021
(MLP-M), GoogleNet (Szegedy et al. 2014) (GNet), ResNet- 50) (R-, 50 ). and DensetNet- (121, ]69] (Huang et al. 2018) (D- [121, 169)) architectures. Due to the LIFAR-100C, we do not include the
limited availability of other kinds of models pre-trained results for it here.
In Fig. 15 we show the relative performance of the probes across all different noise types CIFAR-C dataset.
Using validation set for selecting the probes. We illustrate the performance differences when linear probes are selected using validation set with varying numbers of points in Fig. 16
Validation with 10 datapoints per class
Validation with 100 datapoints per class
20%
ViT
Redbar11
Diference
Difference
15%
10%
10%
5%
5%
0%
0%
Ho
10000A mib
Noise type
Naise type
Figure 16: Comparison of performance differences when linear probes selected on validation set with varying numbers of points.
19

---

Published as a conference paper at ICLR 2025
Rewe god training curves for 1024 po
Layer
Layer
Layer
Layer4
Layer'
Layers
Layer
Layer
0.34
8/0
8.5
94
8.3
6.00
TRADUTEMIA
6.5
6.00
0.90
6.99
6.75
8.7
0.39
25
25
Opcch
Spctt
Coach
Epoch
Esso
Spach
Epech
Figure 13: Accuracies linear probes from last and best-performing layer representations on CIFAR 10C when finetuning 1024 points of OOD data for each noise type.
A.8
CIFAR-100
use pretrained ResNet-18 models on CIFAR-100 from Sonthalia et al. 2004
A.9
SUBPOPULATION SHIFTS
For Waterbirds and CelebA experiments, on different seeds provided the original (Kirichenko et al.) 2023) repository This model was selected based on validation worst group accuracy. For MultiCelebA experiments, the models from (Kim et al. 2023).
of pretrained ERM models dataset trained
A.10 "NATURAL" " DISTRIBUTION SHIFTS ON IMAGENET- 1K
We train probes on ResNets, Bwin-B et al. 202 base version of the capable VIT vari- ant, and ConvNext-B (Liu al. 2022). ViT-B 32, and DensNet Huang al. (2017) architec-
Available from httpp://githhii
Available
from
http:///thhaa.cco//ooaaaaaaaa1ed feature
rewei ght ingeghhghhhhcccciiitt.
20

---

Published as a conference paper at ICLR 2025
Rewelghting ng training curves for 128 points
Layer
Loyer
Layer
Layer4
Layer
Layer 6
Layer
Layer
6.0
8.9
6.75
94
8.30
6.29
TRADUTEMA
6.5
6.75
8.90
6.7%
6.70
0.30
8.7
6.5
25
25
Opech
Spath
Epoch
Esso1
Opach
Epech
Figure 14: Accuracies linear probes from last and best-performing layer representations on CIFAR 10C when finetuning 128 points of OOD data for each noise type.
tures. We divide the considered distribution shifts into two categories: few-class shifts (Cue Conflict and Silbouette), considering 16 classes each, and many-class shifts (Imagenet-A and Imagenet-Bi -R), considering 200 classes each.
21

---

Published as a conference paper at ICLR 2025
Impulse-noise
Pixelate
Speckle-noise
20%
Gaussian-noise
Shot-noise
Gaussian-blur
Accuracy difference (%)
Motion-blur
15%
Snow
Frost
Glass-blur
10%
Defocus-blur
Zoom-blur
Contrast
Fog
5%
Brightness
Saturate
Jpeg-compression
Spatter
0%
Elastic-transform
VIT MLP-Mixer
MLP-Mixer
GoogLeNet
ResNet-18
ResNet-50
DenseNet-121
DenseNet:169
CIFAR-10.2-ter.
CIFAR-10.2-train
Validation
Mean (OOD)
Figure 15: For each model and distribution shift (vertical markers in the plot), relative accuracies for probes across all different noise types CIFAR-C dataset are illustrated (showing gap between best performing last layer probe and the itnermediate one). Mean accuracy across datasets is marked for each model; large gray markers (d, denote performance ID datasets.
22

---

Published as a conference paper at ICLR 2025
Few-shot
Zero-shot
74.3%
75.3%
70
4y
69
65.1
63 8y
63 2%
CIFAR-106 -FFAR-1C Accuracy
80%
60%
40%
20%
60%
05.0
33.3%
32.9%
CIFAR- 10RR
Accuracy
41.0%
27
40%
30.4
20%
0%
5
6
5
6
Layer
Layer
Figure 17: Impact of Depth ILC Performance for CIFAR-10 and CIFAR-100. We evaluate both
zero-shot few-shot settings using ResNet-18. The performance shown across different layers network.
B DETAILS ON ANALYSIS
B. IMPACT OF DEPTH
In Fig. B.I we present additional results on the impact of depth on the performance of ILCs on
CIFAR- 10 and CIFAR-I00 using ResNet-18 model. We observe similar patterns to those in $5.1 (1)
the ILC using pen- con-penultimat layer features (ILCz-2(-)) outperforms last-layer retraining (ILCG-1). and (2) the optimal ILC relies on features from earlier layer.
B.1.1 ID AND OOD PERFORMANCE ACROSS LAYERS
In the main text ($5.1), we presented how OOD performances vary across layers in both zero- and few-shot cases. A natural question arises regarding the relationship between OOD performances across layers and ID accuracies. In this section, conduct experiments to compare ID and OOD accuracies in both settings.
We observe in Fig. 18 that while ID performance generally increases with deeper layers, consistent with previous findings (Alain Bengio 2018), OOD performance does not exhibit the same trend. Specifically, the best berf-performing intermediate layers for zero-shor shot and few-shot scenarios often lie the middle or earlier parts of the network, challenging the assumption that the penultimate layer always offers the most robust features for OOD tasks. This reinforces the utility of intermediate layers over the last layer for better OOD generalization.
B.2 SENSITIVITY ANALYSIS
We explore two ways of measuring sensitivity of features: global, distance-based way, and local, neuron-level measure. In the main text (5.2) we presented results based on global measure.
B.2.1
ADDITIONAL RESULTS ON FEATURE SENSITIVITY
The results shown in Figure 19 further reinforce our findings that intermediate-layer representations exhibit lower sensitivity distribution shifts compared penultimate representations.
B.2.2 LOCAL MEASURE OF FEATURE SENSITIVITY
An alternative way of measuring feature sensitivity OOD shifts is using local, per-neuron measure. We measure the Total Variation Distance (TVD) between ID and OOD data intermediate layers. For each dataset and layer, we compare the distributions formed by the representations of ID and OOD data for each feature across the layer. TVD is defined as: TVD(P, Q) == 1/2 Σ Ex (P) supp(Q) [P(z)
Q(x)). We then average the TVDs across all features for each layer. This method provides measure
23

---

Published as a conference paper at ICLR 2025
CMNIST
1D accuracy accuracy
92.7%
65.1% 74.3%
33.3%
accura
(Zerc-chot)
95.0%
67.5%
52.1% 31.2%
19.7%
93.8%
39.1%--11.9%
Water birds
CelebA
MultiCeleba
98.3%
86.7%
95.8%
80.6% 97.11
46.0%
accuracy
(Zero (horr-choll)
54.4%
94.1%
68.3%
33.4%
84.2%
92.4%
56.0% 94.4%
20.9%
Layer
Layer
Layer
(a) Zero-shot and ID performance across layers.
CMNIST
CIFAR-10
CIFAR-100
securacy accuracy
96.2%
95.9% 92.7% 7%
75.7% 74.3%
45.2%
accuracy
(Fow-shat)
95.0%
67.5%
57.1% 31.2%
24.8%
93.8%
82.7% 42.4%
38.5%-111.%%
4.4%
Waterbirds
CelebA
MultiCeleba
98.3%
94.1% 95.8%
91.0% 97.1%
79.6%
accuracy
91.3%
74.4% 94.1%
84.1% 95.7%
72.7%
94.4%
65.8%
Layer
Layer
Layer
(b) Few-shot and IDD performance across layers.
Figure 18: Performance across layers for ID, and Zero-shot and Few-shot settings. The comparison of ID and OOD performance across layers. While ID performance generally increases with deeper layers OOD performance does not show the same trend. ResNet models used.
CIFAR-10C
CIFAR-100C
3.0
0.30
1.5
0.15
5
6
6
6
8
Layer
Layer
Figure 19: Sensitivity to input perturbation shifts across layers. CIFAR 10C and LLFAR-100G feature sensitivity across layers using ResNet-18.
24

---

Published as a conference paper at ICLR 2025
of how much the individual features of the representations vary with the data distribution. A lower TVD indicates that the representations are less affected by distribution shifts, which suggests that classifiers relying these representations might also be more robust to shifts. This approach allows us to connect the observed transferability of intermediate representations to a quantifiable metric, making the concept of "sensitivity" precise.
Specifically, for each pair (DID, DooD) and layer 1 S 1 < L, the layer maps the input space Jt R4-1 + IRd:, producing a flattened representation with di features. For each feature 1 S is di, we construct distributions (f OOD data, binning these features into 40 bins, and approximating the distributions using histograms over the union of the ranges of ID and OOD features. Finally, we calculate the TVD between the distributions for each feature and average the TVDs across all features for each layer.
and POOD (oon(Ji(X)) (x))::) by computing the features for ID and
For subpopulation shifts, each dataset exhibits group-level shifts. Group labels allow us to measure shift sensitivity for each group. For example, CelebA consists of four groups (a Cartesian product of two groups: Hair-coloz and Gender), and for each group S g < 4. Dib is the training data, and DooD is the validation data belonging to group g. For input-level shifts CIFAR-IOC and CIFAR-100C, for each noise type 1 15 g S 19, Díp comprises features from validation set samples (no shift), and DoDD contains validation data with noise type g. To avoid sample imbalance among groups, we consider only the lowest number of samples among the groups. For Waterbirds, CelebA, and MultiCelebA, we consider 67, 1387, and 40 datapoints, respectively. For CIFAR shifts, we consider all 10,000 datapoints.
We illustrate the TVDs between ID and OOD data for the subpopulation shifts and the input-level shifts in Fig. 20 We find that intermediate layers generally less sensitive to distribution shifts compared to the last layer on subpopulation shifts under most-frequency present groups; underrepresented groups generally exhibit higher TVDs in the last layer compared to the intermediate layers. While intermediate layers also show this pattern, the spread across the TVDs of different groups is smaller. Since the last layer was found be less effective for generalization in all the considered cases, sensitivity to distribution shifts provides potential explanation for its limitations.
Waterbirds
CelebA
MultiCelebA
CIFAR-10C
CIFAR-100C
0.32
0.24
0.4
Minority
Majority
0.4
0.30
0.16
0.3
0.2
0.15
0.08
0.16
0.2
5
5
6
7
5
Layer
Layer
Layer
Layer
Layer
Figure 20: Total variation distances (TVDs) between ID and OOD data. We consider subpopulation shifts (three lefimost plots). where the lines colored according the fraction of samples the group present in the training data, and input-level CIFAR shifts (two rightmost plots).
Layer sensitivity predicts performance. For the input-level shifts, features across layers for CIFAR- 10C and CIFAR-100C exhibit similar TVDs, but the TVDs generally lower for the intermediate layers compared to the last layer. While we have shown in the previous sections that the last layer is less effective for generalization, we additionally show that higher TVD values associated with
lower performance (Fig. 21
CIFAR-10C
CIFAR-100C
Accuracy
0.75
0.50
0.50
0.25
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
Average TVD
Average TVD L6. 0
L5, 0=76.7
L6. 0.95
L8,
0=98.
L5, p=
L8, 0=90.9
Figure 21: Layer sensitivity, More sensitive layers are associated with lower performance on CIFAR- 10C and CIFAR- 100C datasets. Layer 7 is excluded for CIFAR-IOC due to it being near- sear-identical to layer 8.
25

---

Published as a conference paper at ICLR 2025
C ADDITIONAL RESULTS
C.I RELATION TO NEURAL COLLAPSE
We discuss related work by (Li al. 2022) and additionally inspect the relations between OOD gen- eralization and Neural Collapse (NC) (aapyan et al. 2020). Neural Collapse describes phenomenon where class means align along equiangular directions, and features collapse within their respective class means in the last layer of deep networks. This behavior is often quantified using metrics such as NC₁ (Papyan et al. 2020), which measures the alignment of within-class and between-class covariance matrices.
Prelininaries. Following (Papyan et al. 2020). NC1 is defined as:
trace (Zw 21²)
NC1 = K
where Ew E Rdxd and En E Raxd are the within-class and between-class covariance matrices, respectively:
K
1
(r! (r.)) - (()))( T(
Ew
(4)
nk
k=1 i=1
K
ΣB
1 K
(F(K) FF) FGG)
(5)
(k) Here, K is the number of classes, ny is the number of samples in class k, r; is the feature representation of sample in class k ((R) is the mean feature vector for class k and ieg is the global mean of all features.
In its original definition, NC₁ is applied to penultimate-lay layer representations. To study NC across intermediate layers 1, Ew and En can be adopted to compute these statistics layer-wise using rx(),, the representation of input at layer 1.
To approximate NC; behavior layer-wise, particularly under OOD conditions, we use the Class- Distance Normalized Variance (CDNV) metric (Galanti et al. 2021). CDNV is a practical alternative that focuses on class separability and compactness. For a given layer I, CDNV is defined as:
(R,, + Var RR V,R,,
(6)
(R,) i. (1,)1
where:
1 Σ rx),, (R.)
Van
= Σ T1 xEX.
||T(x) - (R.)
(7)
THE XEX;
Here, (x) represents the representation of sample x at layer I, R, denotes the set of representations for class i, and X, is the set of input samples for class i. For each layer I, the final CDNV score is computed as the mean of Vi(R,, R,)) over all pairs of classes + j.
Setup. We compute (R,, R,) across ResNet1 18 layers to characterize NC behavior under distri- bution shifts for CIFAR-10 10 and CIFAR-100 ResNets, evaluating on CIFAR- 10C and CIFAR- 100C, respectively.
We note that this setup differs from Lii et al. 2022). where models were trained on ImageNet- K and evaluated on downstream datasets like CIFAR-10. our case, models are evaluated on the same dataset the training dataset but under distribution shifts of covariates.
Results. We present results in Fig. 22 and Fig. 23 for CIFAR-IOC and CIFAR-10OC, respectively. We also indicate NC1 scores for the validation set of CIFAR-10 to compare how models' representation collapse differs under distribution shifts. We note that in both cases, the model is most collapsed at the penultimate layer (layer& 8), even on the downstream datasets. These results differ from those in (Li et al.) 2022). which suggest that the layer where the model is most collapsed depends on the downstream tasks. Given that the experiments in (Li et al. 2022) were conducted general
26

---

Published as a conference paper at ICLR 2025
pre- vre-trained models (as opposed to task- datasets in our study), we believe this to be the key reason for the observed difference in behavior.
"brightness"
"contrast"
"defocus blur" "elastic transform
"fog"
10
10
10
"frost"
"gaussian blur"
"gaussian noise"
"glass blur"
"impuise noise"
101
10
10-1
"jpeg_cooe compression" "motion blur"
"pixelate"
"saturate"
"shot noise" "
10a
NC²
10"
10-1
"snow"
"spatter"
"speckle noise"
"zoom blur"
"base"
101
NC:
10a
10 1
1
2
3
5 6
7
8
1
2
5 Layer
6
7
1
2 3
4 5
6
2
4
5
6
8
2
3
4 5
Layer
Layer
Layer
Layer
Figure 22: Neural Collapse across layers using ResNet18 CIFAR-10C datasets.
"brightness"
"contrast"
"defocus blur"
"elastic, transform*
"fog"
101
10a
"frost"
"gaussian blur"
"gaussian_noiss
"glass blur"
"impulse, noise" "
101
NC,
10a
"jpeg_coome compression" "motion blur"
"pixelate"
"saturate"
Schot_noise"
10
NC:
10"
"snow"
"spatter"
noise"
"zoom blur"
"base"
10
10"
4
5
6
4
3
4
7
8
2
3
4
8
1
2
4
7
8
Layer
Layer
Layer
Layer
Layer
Figure 23: Neural Collapse across layers using ResNet18 CIFAR-100C datasets.
27

---

Published as a conference paper at ICLR 2025
C.2 NON-LINEAR ILCs
In this study, we have restricted intermediate-layer classifiers to linear probes to keep the complexity and computational costs low, ensuring the method remains practical. However, non-linear classifiers might better capture complex representations at intermediate layers, potentially improving perfor- mance. Moreover, the performance of non- non-linear probes can help assess whether the representations contain learnable information that is otherwise inaccessible to linear probes, providing further insights into the informativeness of these representations.
Setup. To investigate this, we train a one- mee-hidden-laye layer MLP with 512 neurons and ReLU activation, which feeds into the linear classifier. We conduct few-shot experiments on CIFAR- and CIFAR- 100C using ResNerl8 8 and VIT. The setup and hyperparameter search follow the same protocol as $4.. We repeat the experiment with three random initializations of the classifier parameters to ensure robustness of the results.
Results. Non-linear probes improve performance on both CIFAR-I0 and CIFAR-I00 100, particularly in the third-to-last layer, as shown in Tables SJndes This trend highlights the ability of non-linear probes to extract more complex patterns. For the last layer, the performance varies significantly across runs, with marginal to non-existent improvements average. This variability suggests that, while some OOD specific information might occasionally accessible, the representations are generally insufficiently robust for OOD generalization and do not contain rich information content as intermediate layers.
Table 5: Results for ResNet18 on CIFAR-10 and CIFAR-100.
Layer ID
CIFAR-10 (%)
CIFAR-100 (%)
Linear Probes
Non-Lineen Probes
Linear Probes
Non-Linear Probes
L
74.57%
75.07%
30.40%
34.10%
L 3
75.67%
75.71%
42.71%
41.16%
L 2
70.75%
71.27%
45.01%
45.83%
L
70.26%
70.95%
44.04%
44.25%
Table 6: Results for VIT CIFAR-10 and CIFAR-100.
Layer ID
CIFAR-10 (%)
CIFAR-100 (%)
Linear Probes
Non-Linear Probes
Linear Probes
Non-Linear Probes
L - 4
73.54%
75.50%
41.96%
43.35%
L -
75.19%
77.12%
41.81%
43.31%
L
76.04%
77.07%
47.46%
47.82%
L 74.71%
74.48%
47.69%
47.47%
C.3 IMPACT OF FEATURE DIMENSIONALITY ON PERFORMANCE
In all of our experiments, we used features extracted at each depth of a neural network. For CNNs, this resulted in features whose dimensionality changed across layers, while for ViTs, the features had fixed dimensionality across layers. In the case of CNNs, the feature dimensionality progressively decreased. For instance, for an imput image of dimension produces features of dimensionality 8, 192 at layer 5, 4, 096 at layer 6, 2, 048 at layer 7, and 512 at layer& 8. In all of our experiments, linear probes were trained on the raw features (after flattening them), without applying any pooling techniques.
32 32, in CIFAR- 10, ResNet18
Given the large variation in feature dimensionality across layers, a potential concern that the gains we observed in 44.2 and $4.3 arise from differences in feature dimensionality alone. We argue that this is not the case for two reasons. (1) For ILCs to succeed, the features in intermediate layers must be inherently informative of the task. If such features exist at these layers, dimensionality alone should not significantly influence performance (either the useful features are present or they are not). (2) Feature dimensionality does not explain the observed pattern in in-distribution (ID) performance,
28

---

to evaluate Published as a conference paper at ICLR 2025
where the highest performance is almost always achieved using last-layer features. As shown in (B...I] ID performance peaks, or is near the peak, when using penultimate layer representations in the classifier 1LCr.-1 whose feature dimensionality is the smallest.
To validate this reasoning, we conduct an experiment to investigate how standardising feature dimensionality using PCA influences performance CIFAR-10 with ResNet18.
Setup. Feature representations across layers were standardised using PCA. For each layer i, we extracted training representations R. IR Nxd, where N is the number of training samples and d; is the original feature dimensionality of the layer. These representations were centred as R; R. mean (R,), and the top 512 PCA directions V, R4, x512 were computed. For test samples, representations (x) E Rd were centred using the training mean and projected onto the PCA basis as:
r
(x)
(r,(x) mean(R.) VV 5R
This ensured that the dimensionality all layer representations was fixed to 512. Linear probes were then trained and evaluated these fixedddimensional representations in few-shot setup.
Table 7: Few-shot accuracy (%) ) on CIEAR-10C with ResNet18 before and after PCA. Dimensionality after PCA was fixed to 512 for all layers.
Layer
Dim. (Before PCA)
Dim. (After PCA)
Acc. (Before PCA)
Acc. (After PCA)
192
512
74.3%
71.4%
6
4, 096
512
75.3%
74.6%
7
048
512
70.4%
71.3%
8
512
512
69.9%
69.9%
Results. Table shows the results. Performance trends remained consistent after PCA, with layer achieving the highest accuracy in both setups. The minor differences in accuracy before and after PCA suggest that dimensionality alone does not drive the trends observed in layer-wise performance.
C.4 LAYER-WISS WISE TRANSFERABILITY ACROSS DISTRIBUTION SHIFTS
To better understand the role of individual layers in handling distribution shifts, we evaluate their cross-dataset and cross-shift performance while keeping the model architecture fixed. Specifically, we analyse the transferability of layers in ResNet18 by examining their performance on CIFAR-I0C and CIFAR-100C under various corruptions in the few-shot setting. This experiment assesses how well features from a specific layer generalise across datasets with similar distribution shifts and measures the extent to which layers transfer across distribution shifts and datasets.
Setup. For each corruption type CIFAR-I0C, we evaluate the performance of layers E f3..... 8} when transferring to the same corruption type in CIFAR- Linear probes are trained separately on features extracted from CIFAR-10C and CIFAR-100C, following the setup described in $4.2. The best probes are selected based on the validation split. For any given dataset, distribution shift, and layer, we compare how the probes perform on CIFAR- versus transferability.
29

---

Published as a conference paper at ICLR 2025
"gaussian_ _
"brightness"
"contrast"
"defocus blur"
0.35
0.65
0.42
0.51
CIFAR100 CIFAR100
0.14
0.13
0.10
0.14
0.53
0.58
0.63
0.68
0.58
0.69
0.79
0.
0.37
0.47
0.57
0.66
0.56
0.64
0.73
0.82
"elastic transfom"
"fag"
"frost"
"gaussian blur"
0.56
0.52
0.42
0.38
0.38
0.35
0.28
0.25
0.25
0.25
0.14
0.09
0.12
0.14
"glass blur"
"impulse "
"jpeg compression"
"mation, blur"
CIFAR100
0.40
0.25
0.53
0.54
0.15
0.10
0.49
0.56
0.63
0.70
0.33
0.40
0.46
0.53
0.56
0.65
0.74
0.83
0.55
0.64
0.73
0.81
"pixelate"
"saturate"
"shot_nols"
"snow"
CIFAR100
0.48
0.57
0.37
0.42
0.29
0.39
0.25
0.28
0.21
0.26
0.14
0.13
0.14
0.12
"spatter"
"speckle_noise
"20om_ blur"
CIFAR10
CIFAR100
0.48
0.35
0.56
0.13
0.13
0.52
0.61
0.70
0.79
0.53
0.59
0.65
0.70
0.56
0.66
0.76
0.86
CIFAR10
CIFAR10
CIFAR10
Figure 24: Layer transferability between CIFAR-10C and CIFAR- ResNet18. Each scatter plot represents a specific corruption type, with the x- x-axis showing the accuracy on CIFAR 1OC y-axis showing the accuracy on CIFAR- 100C. Points denote the performance of individual layers E 83.
Results. The results are summarised in Figure 24 We highlight two key observations regarding layer transferability across datasets and distribution shifts:
(1) Pen-penultimate layer consistently outperforms penultimate layer. For each corruption, the pen-penultimate layer achieves higher performance than the penultimate layer both CIFAR-IOC and CI0AR-100C.
(2) Dataset-specific optimal layers. The layer achieving the highest performance on CIFAR- is often not the best- pest-performing layer on CIFAR 100C for the same corruption. This suggests that while layers transfer reasonably well, certain layers may provide features that are better suited for specific dataset, even under similar distribution shifts.
This analysis shows that transferable information is distributed non-uniformly across layers and varies by dataset, even with the same architecture, highlighting the importance of fine- tne-grained layer evaluations under distribution shifts.
30


Reviews:
[{"review_id": "gG3Y8uJwFN", "reviewer": "Reviewer_JuAB", "summary": "This paper investigates the use of intermediate layer linear probing classifiers to enhance a model's performance on out-of-distribution (OOD) datasets, where the test data distribution differs significantly from the training data distribution. People usually assume that the network can act as a combination of general feature extraction(early layers) plus in-domain task specialization(later layers). The method builds on the assumption that not only the the last classifier specializes in the in-domain distribution, last few layers in the backbone beyond the final classifier also focus on the in-domain distribution. So it propose to place the probing classifier at an intermediate layer to improve transferability and adapt to OOD settings", "questions": "Based on the weakness I have the following questions:\n\n1. Could you clarify how \"few-shot\" is defined in this context? The use of half the CMNIST test dataset may diverge from traditional definitions of few-shot, which usually refer to very limited samples (e.g., 1–5).\n\n2. Could you provide more details on why zero-shot probing—where only in-domain data is used—seems to enhance OOD performance? This mechanism isn’t entirely clear, as one would generally expect some OOD data to be necessary for adapting to distribution shifts.\n\n3. Has the team considered testing more complex classifiers at the intermediate layer, such as a two-layer MLP? Or add an adaptor between backbone and the linear probing?  Since pruning the intermediate layers may omit useful deep features, a more complex classifier could potentially bridge this gap and enhance OOD results.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper includes comprehensive experiments demonstrating that intermediate layer probing consistently outperforms last-layer probing in OOD scenarios. This is shown to hold across various network architectures and datasets.\n\n2. By investigating intermediate layer probing, the paper provides a theoretical perspective on measuring distribution shifts within network representations", "weaknesses": "1. The definition of “few-shot” may differ from conventional standards, as it uses a significant portion of the test dataset (e.g., half of the CMNIST test samples) rather than the typically small sample sizes seen in few-shot learning. This discrepancy may limit the generalizability of the results.\n\n2. I'm confused that why the zero-shot probing setting can serve as a method to improve the OOD performance. I thought that when trained on in-domain data alone, the OOD performance would not improve since no OOD data is used to train the probing classifier. Would the zero-shot setting can improve the OOD performance?\n\n3. The use of a single-layer linear classifier at an intermediate layer may restrict its ability to capture more complex feature relationships necessary for challenging OOD tasks. Incorporating a more complex classifier, such as a multi-layer perceptron (MLP) or adaptors, could potentially improve performance on these tasks.\n\n4. The main assumption aligns closely with [1], which suggests that layer-wise neural collapse (NC) progresses through two distinct phases in downstream tasks: Phase 1, where representations exhibit a progressively decreasing trend, capturing universal feature mappings; and Phase 2, characterized by a progressively increasing trend, focusing on task-specific feature mappings. A comparison with this prior work would provide valuable insights into how the proposed method builds upon or diverges from these observed phases.\n\n[1] Li, X., Liu, S., Zhou, J., Lu, X., Fernandez-Granda, C., Zhu, Z., & Qu, Q. (2022). Principled and efficient transfer learning of deep models via neural collapse. arXiv preprint arXiv:2212.12206.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8v4J6xGZcc", "reviewer": "Reviewer_TgHf", "summary": "The authors explore whether the intermediate layers offer better out-of-distribution (OOD) generalisation. They found that in a number of settings, including zero- and few-shot, they do and demonstrate the limits of penultimate layer representation utility. Most interestingly, the authors claim that ‘earlier-layer representations without fine-tuning on the target distribution often fare competitively with the last-layer retraining on the target distribution’.\nThe contribution of the paper is empirical, and the authors support the claims with extensive number of experiments.", "questions": "1. “Line 198-199: using  data that can be either ID or OOD, depending on whether  the setup is zero-shot or few-shot.” Not sure I get this phrase, why would the data be selected ID for the zero-shot scenario and OOD for the few-shot scenario? Shouldn’t it be both ID and OOD for both cases (or just OOD)?\n2. The biggest question and concern is the inclusion criteria for both datasets and the models. The dataset performance beyond most-known datasets such as CIFAR-10 and CIFAR-100 may differ not least due to the fine-grained nature of some of these datasets. Figure 7 from Tobaben et al (2023), for example, chose to include VTAB-1k datasets to test the performance on the different datasets. On the inclusion of the models, on one hand, it seems to give a fair share of ViTs and ResNets, however it would be great to spell out the inclusion criteria: why did the authors choose this particular selection of the model? Separate subsection in  Section 4 seems to be an appropriate place for this. \n3. The authors can argue that it’s not within the scope, so it would be a fair play, however, it would be interesting to see the following. It appears that the models have been trained on the clear target dataset (i.e., the authors  evaluate the OOD performance of the CIFAR-100 trained model), and I wonder if they tried whether these findings generalise for models, pre-trained on generic large-scale data (i.e. ImageNet-pretrained or pretrained models such as DinoV2 or other varieties of pretrained ViT)? This may be useful in finding out whether the conclusion of the paper remain valid for the scenarios when  \n\nTobaben et al (2023) On the Efficacy of Differentially Private Few-shot Image Classification, TMLR 2023", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Pros:\n- Significance: The question of generalisation, especially to the zero-shot scenarios, is an important one. While I think it’s been tremendous work done, s I discuss in comment 2, it is important to discuss a number of limitations such as selection of evaluation dataset,\n- Correctness: I haven’t spotted any incorrect statements in the paper\n- Motivation: the paper challenges the conventional wisdom of using the last layer for zero and few-shot learning, and I believe it’s a great motivation\n- Reproducibility: I’ve checked the experimental setting and it looks reproducible to me. I would also highlight that the description of the experiments is exceptionally clear\n- Clarity: the paper is very clearly written (apart from a couple of comments below)", "weaknesses": "Cons:\n- Clarity and limitations: a few questions on the experimental setting", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kfiYwrTYoo", "reviewer": "Reviewer_ncZg", "summary": "This paper shows that in ResNets and ViTs, intermediate layers classification is more robust to the distribution shifts of the out-of-distribution samples (OODs). In particular it shows that the features of the pen-pen-ultimate layer consistently outperform the features of the pen-ultimate layer, when it comes to the zero-shot and few shot cases. These results are interesting, as they contradict the current belief that it is enough to retrain the last layer, that is the classification layer. It is assumed that each layer computes interesting invariants of an input image, and this paper raises the possibility that some of these invariants get lost in downstream layers.", "questions": "In principle, the features of intermediate layers, are propagated down all the way to the pen-ultimate layer, through skip connections, in both ResNets and ViTs. However, they are added before every layer with the output of the previous layer. \n\nIs this the reason of why generalization ability decreases? \n\nFrom the point of view of back-propagation the addition does not matter, as the derivative of a sum is the sum of the derivatives. However, this addition may clutter the features?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is a very intriguing paper, that raises the question of what happens in the last layers, that overcomes generalization to OOD samples?", "weaknesses": "While analyzing the depth and the sensitivity as important factors in generalization ability, it would be interesting to understand what happens in the last layers that reduces generalizability?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "paSEucgPQJ", "reviewer": "Reviewer_G9DE", "summary": "The paper investigates how intermediate layers in deep neural networks (DNNs) can improve out-of-distribution (OOD) generalization, challenging the common practice of relying solely on the last layer's representations. The authors use the notion of Intermediate Layer Classifiers, which consist of linear probes attached to each intermediate layer of the network but trained specifically to perform OOD classification. Two scenarios as considered: one \"few-shot\", in which few OOD samples are available for training, and one \"zero-shot\", where no OOD samples are available, under several types of distribution shifts, datasets, and architectures. They conclude that in both cases training classifiers on intermediate layer representations tends to perform better than last-layer retraining, and suggest that one of the reasons for this phenomenon is that intermediate representations are less sensitive to distribution shifts.", "questions": "- __Figure 3:__ which ResNet was used to produce the bar plot?\n- __Figure 4:__ which ResNets were used to produce these plots? Was any trend observed when comparing the different ResNet sizes?\n- __Section 4.3:__ can we really call \"zero-shot\" a setting in which the \"OOD\" data is composed of ID samples with added noise? That is a standard modification used in data augmentation (used to increase the number of ID samples), for example, so such a \"distribution shift\" seems extremely mild to be considered zero-shot. I would have liked to see a discussion of the different shifts chosen compare to one another in terms of difficulty.\n- Did the authors find that any particular architecture had intermediate representations consistently outperforming last-layer retraining?\n- Did the best layer for a particular dataset seem to agree with the best layer for other datasets, for the same model?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is very well-written.\n- The paper assesses intermediate-layer performance across a variety of tasks, datasets, and architectures, which strengthens the generalizability of the findings.\n- Interesting discussion on why intermediate layers might generalize better to OOD data, including their reduced sensitivity to distribution shifts and improved data efficiency.\n- By demonstrating that intermediate layers can achieve competitive results even in few-shot or zero-shot settings, this work provides practical benefits, especially for applications where OOD data is scarce or unavailable.", "weaknesses": "__Sensitivity Analysis:__ although the sensitivity analysis aims at showing the stability of intermediate-layer features, it lacks sufficient depth on why some intermediate layers are more stable than others in different types of shifts, particularly for different architectures. The inclusion of a summary of any trends observed across architectures/datasets/shifts would be helpful.\n\n__Figure 10:__ The point made in the description of Figure 10 is not clear from the plots shown, especially since it is not clear how many points occlude one another in the scatter plots. What dataset was used here, and what model? The quantitative analysis is also confusing: although the authors say that an increasing separation exists in earlier layers, I believe this only holds for the minority groups. Perhaps the authors should clarify this.\n\n__Terminology:__ I found the terminology a bit confusing when dealing with training points vs. testing points vs. probe points vs. OOD points. Namely, I had trouble following which sets contained OOD points vs. ID points, especially in the zero-shot section. It would be advisable to always point out which points are OOD.\n\n__Novelty:__  The literature reviewed for related work seems superficial. A notable example of a paper missing from the references is:\n\n[1] Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). \"How transferable are features in deep neural networks?\"_Advances in Neural Information Processing Systems_, 27. https://proceedings.neurips.cc/paper_files/paper/2014/hash/375c71349b295fbe2dcdca9206f20a06-Abstract.html\n\nwhich explored a method for quantifying the transferability of features from each layer of a deep network, including their applicability to shifts in the data (e.g., different subsets, and even different classes).\n\nFurthermore, the novelty of this contribution is somewhat diminished given that an earlier paper (accepted at last year's ICLR) has also proposed evaluating intermediate layers’ effectiveness in generalization for OOD samples:\n\n[2] Gerritz et al. (2024) \"Zero-shot generalization across architectures for visual classification.\" _The Second Tiny Papers Track at ICLR 2024_, https://openreview.net/forum?id=orYMrUv7eu\n\nIn fact, an extended version of that paper (from the same group) also used linear probes to quantify the out-of-sample accuracy of intermediate representations:\n\n[3] Dyballa et al. (2024). A separability-based approach to quantifying generalization: which layer is best?. _arXiv preprint arXiv:2405.01524_, https://arxiv.org/abs/2405.01524\n\nThis paper would certainly benefit from acknowledging that similar observations have been previously made when performing category discovery on unseen classes (as opposed to the distributional shifts here studied).\n\n__Frozen weights:__ The fact that the authors only considered frozen, pre-trained weights in all scenarios studied is somewhat unrealistic or at least incomplete. In any practical application of this method, if it is feasible to fine-tune the pretrained model on one's own data, then that will certainly be preferable. See reference [1] for an example in which both cases (frozen weights vs fine-tuned) are considered. Specifically for this paper, it would be interesting to verify whether the comparisons between penultimate layer and intermediate layer outputs hold in that scenario.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BsAQI3m2vZ", "reviewer": "Reviewer_Nh8y", "summary": "This paper questions the use of last-layer representations for out-of-distribution (OOD) generalization due to their sensitivity to data distribution shifts. Instead, by introducing intermediate layer classifiers (ILC), the paper shows that intermediate layer representations often outperform penultimate layer representations in generalization.", "questions": "1. Could the authors provide more explanations about the feasibility of using the sensitivity score as defined in the paper? Why is this score appropriate for characterizing the sensitivity?\n2. All the experiments are based on the task of image classification. What will the results become when dealing with other tasks besides image classification?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and easy to follow. The observation is interesting that intermediate layers may perform better than the last layer in generalization. And, the paper introduces a new metric named \"sensitivity score\" to measure the sensitivity of each layer to distribution shifts.", "weaknesses": "1. The paper discusses a concept called \"information content\", but it lacks some detailed explanation. The authors need to demonstrate why the accuracy on OOD tasks can characterize \"information content\".\n2. The potential impact is not so clear. The observation is interesting, but it may be difficult to find the \"best\" layer in practice. \n3. It is better to provide more theoretical backups to support the experimental findings.\n\nSmall typos: \"atop\" (line 469); \"in earlier layers\" (line 497); \"$\\pi$\" not defined (line 500)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Txl2m1wpg4", "reviewer": "Reviewer_P4oy", "summary": "This paper demonstrates that leveraging intermediate features outperforms conventional approaches relying on last-layer features for OOD generalization. The claim is substantiated through extensive experiments across diverse datasets, varying data quantities, model architectures, and types of distribution shift. Additional analysis on feature sensitivity further reveals that intermediate features exhibit greater robustness to distribution shifts.", "questions": "Please see the Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and clearly presented.\n* The experiments are comprehensive, covering multiple aspects.\n* The analysis of feature sensitivity is interesting.", "weaknesses": "* The most convincing experimental results supporting the paper’s claim are observed on CNNs, whereas the performance gains on (currently) more widely used transformer-based architectures are less pronounced, with intermediate features providing only marginal improvement over last-layer features. It may worth discussing the potential reason for this performance disparity. \n* More critically, while the paper offers detailed descriptions of the data and model setups, it lacks a clear feature extraction protocol. For instance, when using an intermediate feature with dimensions $256 \\times 4 \\times 4$, do the authors apply pooling before probing or directly flatten the representation? This question is important since ConvNets almost always have a feature dimension change across layers and this may contribute to the performance difference between intermediate features and last-layer features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)