Don’t forget the nullspace! Nullspace occupancy as a mechanism for out of distribution failure
Authors: Daksh Idnani
Vivek Madan
Naman Goyal
David J. Schwab
Shanmukha Ramakrishna Vedantam
Venue: ICLR 2023 poster (2023)

Abstract: Out of distribution (OoD) generalization has received considerable interest in recent years. In this work, we identify a particular failure mode of OoD generalization for discriminative classifiers that is based on test data (from a new domain) lying in the nullspace of features learnt from source data. We demonstrate the existence of this failure mode across multiple networks trained across RotatedMNIST, PACS, TerraIncognita, DomainNet and ImageNet-R datasets. We then study different choices for characterizing the feature space and show that projecting intermediate representations onto the span of directions that obtain maximum training accuracy provides consistent improvements in OoD performance. Finally, we show that such nullspace behavior also provides an insight into neural networks trained on poisoned data. We hope our work galvanizes interest in the relationship between the nullspace occupancy failure mode and generalization.

Published as a conference paper at ICLR 2023
DON'T FORGET THE NULLSPACE! NULLSPACE OCCU- PANCY AS A MECHANISM FOR OUT OF DISTRIBUTION
FAILURE
Daksh Idnani Meta (FAIR)
Vivek Madan Amazon
Naman Goyal Meta (FAIR)
David J. Schwab Meta (CTRL)
Ramakrishna Vedantam Meta (FAIR)
ABSTRACT
Out of distribution (OoD) generalization has received considerable interest in recent years. In this work, we identify a particular failure mode of OoD generalization for discriminative classifiers that is based on test data (from new domain) lying in the nullspace of features learnt from source data. We demonstrate the existence this failure mode across multiple networks trained across RotatedMNIST, PACS, Terralncognita, DomainNet and ImageNet-R datasets. We then study different choices for characterizing the feature space and show that projecting intermediate representations onto the span of directions that obtain maximum training accuracy provides consistent improvements in OoD performance. Finally, we show that such nullspace behavior also provides an insight into neural networks trained on poisoned data. We hope our work galvanizes interest in the relationship between the nullspace occupancy failure mode and generalization.
INTRODUCTION
Neural networks often succeed in learning rich function approximators that generalize remarkably well to the distribution they are trained on, but are often brittle when exposed to inputs that come
from different distribution (Gulrajani & Lopez-Pax Paz 2020 With rapid adoption of neural networks to various safety critical applications such as autonomous driving, healthcare etc. more attention is
being paid the question of robustness under domain shift (Alcorn al., 2018 Dai & Van Gool
2018 AlBadawy el al. 2018).
Recent findings from Huh et al. 2021 hint that overparameterizee,, deep neural networks biased
to Iearn functions with (approximate) low-rank covariance structure and posit that this related to the phenomenon of implicit regularization (Galanti & Poggio 2022) that has been used to explain in-distribution generalization of deep networks.
might be
How might such low-rank structure relate to out- aut-of-distributtin generalization? As a simple thought experiment, consider a setting where training data Derain is embedded in a three dimensional space (V1, V2, V3) that exhibits variance only along the first two dimensions (fig. (left) (with Va = 1) Let us train neural network So on this data using a loss functional L(F, Drrain). Since va does not contribute to any reduction in training error, standard empirical risk minimization (ERM) (Vapnik) 1999) training need not differentiate between functions f which handle v3 in different ways.
Now consider an out-of-distribtiion (OoD) dataset which has the same structure as the original dataset along V1 and V2, but the value of Va now has a different value, e.g. 10. In this case, one would incur an error (fig. right) if one learns a function ff where f(.. V3 = 1) + f(.... V3 = 10). Thus, the low-rank simplicity bias, while beneficial for IID generalization (Huh et al. 2021) can potentially cause issues for OoD generalization. In cases where removing the "additional" features observed at
test time improves performance (such in fig. we say that the network incurs nullspace error and call the failure mode as "nullspace occupancy"
This is special of Huh 2021) where the third eigenvalue O, instead of heing very small

---

Published as a conference paper at ICLR 2023
Trann i Wel
as w-ll
When diagnosing nullspace oceupancy related failure, it is important to properly choose the representational basis. Our key technical con- tribution is combining notions of variation in training data with utility for the downstream network f in order to identify the most impor- tant directions for projection. We formalize this as an optimization problem that we solve via projected gradient descent onto orthogonal ma- trices (Kiani et al. 2022).
-33
-81
-12
110
10 03 10
Figure I: Illustration of nullspace failure. Left: Training data with variation V1, V2 but no vari- ation in V3 1. Black line: Decision boundary learnt by a hidden layer MLP function ʃ with inputs (V1: V2. Vz) visualized with V3 = 1. Right: Decision boundary of the same classifier evaluated on the plane Va 10. f is sensitive to V3. causing nullspace error (red box) on test data.
Experimentally, we demonstrate the existence of nullspace occupancy related performance degradations across different architectures and datasets evaluated on the DomainBed bench- mark (table 2}. This empirically establishes that nullspace occupancy is an issue for neural net- works in OoD settings and suggests further performance improvements by mitigating it. We take first steps towards showing this in practice in the
setting from Gulrajani &
Lopez-Paz (2020) to improve OoD performance on DomainBed.
Overall, our contributions
follows:
We identify nullspace occupancy based failure mode for OoD generalization
We demonstrate that this failure mode exists for models trained using ERM on DomainBed
We observe that selecting few projecting directions with high training accuracy yields the maximum potential improvements using this approach
Interestingly, we also find that in data poisoning setting (Huang et al. 2020). the network
exploits this nullspace occupancy phenomenon to learn a poorly generalizing classifier (section
METHODS
2.1 FEATURE PROJECTION MECHANICS
We work in the standard multi-class classification setting with inputs and labels y. Let z = g(x) e R be the features extracted in an intermediate layer of the network / using an encoder g. and μ = N >E be the mean of the features z; extracted on a training dataset with N datapoints. Let f be the classification function that maps a feature z to the logits. Finally, let V E R R = [VI,,... vxl be an orthonormal matrix, and a rank m subspace of V is Vm Kxxm V..,.., vm].
Given a test datapoint &, with corresponding extracted feature 2, one can project the datapoint onto a basis Vm of rank m as follows:
Classiher from layer++ tol.
m
Vo V p1) +
V
For a convolutional network, we perform projection only along the channel dimension of the featurization. That is, for intermediate representation with n. channels and spatial size H x W, we consider a set of features Vn. of dimensionality n.. For each spatial location (i, [H] (W), we project
Encoder till layert
g
a vector Zij as follows:
2. Vm VT (2:j Mig) + Pij
Figure 2: Our meth- ods add projection layers (orange) to an existing pretrained network with an en- coder go (till layer 1) and downstream classifier f.
2.2 WHAT IS THE RIGHT CHOICE OF BASIS V?
We aim to find a basis V that leads to the highest decrease in the training loss Lirain as we increase the rank m. This ensures that we cover the most important directions for the loss functional L(f. Dtrain). Intuitively, this incorporates both notions of training sensitivity as well feature spread, since directions with a lot of spread of training data and where the function has high sensitivity would also decrease the training loss.
2

---

Published as a conference paper at ICLR 2023
We propose directly learn the projection matrix that identifies the most parsimonious V based on the training set. The key idea simple: we initialize the basis with the principal component basis computed over the training set, i.e. V Vpcs then denoting μ the mean of the features over the training dataset (as above) we solve the following optimization problem, using a projective approach for optimizing unitary matrices (Kiani al. 2022):
K
Dapaial
miny
Σ m=1
Σ
faaain Crrain(S(V VT. (z, - (4) + 14), 34)
(3)
subject to the unitary constraint VT V Here, the training loss Ltrain is cross entropy. For convolutional network, perform analogous projections along the channel dimension of the featurization. we consider the training accuracy versus number of components m, optimizing this equation will yield the basis which maximizes area under the curve. As result, for any given target accuracy, we find the smallest value of m that achieves that level of training accuracy, thus learning the most parsimonious basis.
2.3 BASELINE CHOICES OF BASES V
We consider ablations on the design of the basis V that capture different intuitive notions of what one considers to be important when choosing the features to project onto (see table TT. We explain each more detail below:
Principal Components Analysis: Given the feature matrix Z E R RNNK computed over the training data Dtrains one can compute the principal components analysis (PCA) (Murphy 20022) of Z and use it as the basis V. Intuitively, this captures the directions in the feature space which have the most variance the training data, but might not capture which directions are used by the downstream function ( table D) As we validate in section 4 considering both is important for diagnosing nullspace occupancy related failures.
Function Sensitivity: The Jacobian matrix codes the sensitivity of the classification func- tion f to varying input features 2,. We derive a global feature set (independent a particu- lar z,) via the columns of the Jacobian. Intu- itively, this gives us set of directions which maximally explains the sensitivity of the func- tion in the space V. Concretely, we consider the matrix G of samples (f(2))
Approach
Uses f
Uses g
Uses Ltrain
Random
X
X
PCA
X
/
X
Low-rank W1+1
X
X
Jacobian
/
X
Optimized Basis
/
de, } where each sample is the Jacobian vector of one out- put logit E ) (chosen by sampling from the output distribution of the classifier ʃ). We do this since backpropagating through each logit (for each training datapoint) in turn is too ex- pensive, Next, we decompose G = U- VT using singular value decomposition (SVD), and use the columns E K of the right-singular matrix V as the feature set. This gives us a set of directions in the input space which explain the dominant directions in which the function varies (but does not capture directions which necessarily have variance in the training dataset Dtraen).
Table 1: Design space of the different bases and ap- proaches considered in the paper. is the network encoder (till layer E) and f is the downstream classi- fier as illustrated in fig. 2 Ltrain is the training loss.
Random Basis: We start with (O, I), where I the identity matrix in K and then use Gram-Schmiae orthonormalization to get full rank random matrix V for truncating features Vn..
Low Rank Linear Layer: Consider the linear embedding (or convolutional kernel) Wi in the g(x). We consider here low-rank approximations to this weight matrix to study if any potential OoD improvements come from nullspace phenomenon or some other kind of capacity control. Following (Kolda & Bader 2009), consider the SVD of W411 A T where S is a diagonal matrix of singular values diag((s). ... ,s]) such that $1 2 82 2 ... 2 8,, and is the rank. A low-rank approximation W4+1 can then be easily found by truncating s to top
layer ( + proceeding
3

---

Published as a conference paper at ICLR 2023
m singular values, that is, Sm diag([su:" 8m ..... (0...,00)))) .0] and the corresponding low rank W = A Bm
EXPERIMENTAL SETUP
Training Protocol: We evaluate on a number of domain generalization datasets provided part of the DomainBed (Gulrajani & Lopez-Paz 2020) benchmark. Specifically, we consider RotatedMNIST (Ghifary et al.j 2013), PACS (Li et al., 2017b). TerraIncogmita (Beery al. 2018). NemainNet (Peng et 2019). and Imagenet-R (Hendrycks et al. (021). We train ERMs using the hyperparameter selection strategy pioneered by (Gulrajani & Lopez- Paz 2020). where we train a large number of ERMs using random hyperparameter search, pick the best on IID validation and use it to report OOD test accuracy repeating this process for multiple independent trials. On RotaTedMNIST we consider two differences to the standard DomainBed protocol: 1) train individual domains and evaluate on all others (to emphasize the fact that our method conceptually does not need environment annotations), and 2) we include search over weight decay (which is fixed to for RotAtedMNIST in (Gulrajani & Lopez-Pat (0200) since weight decay might be related to smoothness of f and nullspace occupancy. Following DomainBed, we use ResNet50 for all experiments on PACS, Terralncognita, DomainNet, and ImageNet-R, and small CNN for experiments on Rotated-MINIST. We note that both DomainBed and ImageNet-R are released under MIT license. To our knowledge, none of these datasets contain personally identifiable information or offensive content.
OOD Accuracy Oracle: Given trained networks f (fig. 2 our objective is to validate whether OOD accuracy improves when we project out the nullspace components for a test data point. Each of the datasets above has multiple train and test environment splits in DomainBed. For the best ERM network that DomainBed yields for a given set of training evironments, we sweep through all the layers and for each layer, we sweep through different values of m, perform the projection and note the peak performance improvement achieved. We then average this number across multiple choices of the training environments (each of which corresponds to different network) to obtain the "oracle" accuracy (o) improvement. Following DomainBed methodology, we repeat the entire process for independent trials, and report the mean oracle accuracy achieved. In another experiment, we sweep through all the layers, but choose the value of m via. some heuristic that can be computed via. Drrain In such cases we call the oracle improvement "layer-oracle" accuracy improvement is an oracle that has access to the ideal layer. The OOD accuracy oracle provides an upper bound to the improvement achievable by our feature projection method, and is indicative of the amount of generalization failure explained by nullspace failure.
lo, since
Leave-one-out evaluation: A limitation of the oracle evaluation method that there no way to pick the correct layer in which to perform projection. Towards obtaining a practically achievable improvement, we perform an additional experiment where we select the layer / and number of components m based highest accuracy on held-out training domain. Our methodology for PACS, TerraIncognita, and DomainNet datasets is as follows: given a test domain, we hold out one training domain for selecting projection hyperparameters and train network using ERM on the remaining training domains. We fit bases on the remaining domains, select the layer and number of components m get the highest achievable improvement on the held-out domain, and evaluate performance on the test domain. We train a network on every combination of training and held-out domain, and compute the mean accuracy on the test domain over networks trained with different held-out domains. We repeat this entire process for independent trials, and report mean accuracy and standard error. For Motated-MNIS'" we train networks on O-degree data, select the layer and number of components on 45- degree data, and evaluate on remaining domains.
4 RESULTS
Does nullspace occupancy relate to OoD failure? We first illustrate (fig.) 3 how the different choices of bases affect accuracy of the pretrained ERM models when performing projection (for randomly chosen network and an oracle choice of layer for the RotatedMNIST. PACS, and TerraIncognita datasets, see Appendix for similar plot on DomainNet). First, notice that projecting with random
4

---

Published as a conference paper at ICLR 2023
Improvemenn in 0.50 accuaayy
moad balr
Jaczzen 2e13
Ozor and ter
0.85
Improveeenncn 0:00
0.05
Improvament.inn
-232
+006
-405
26
289
400
500
5006
Number compoments
companants (layer
Projecting test OoD activations onto subspace of rank m (x-axis) improves OoD performance (y-axis). For each dataset- - Rotated-MNIST (left), PACS (middle), and Terralncognita (right) - we pick one illustrative network out of 12 networks and the layer which best explains the nullspace occupancy related failure mode.
Figure 3: Nullspace occupancy relates to OoD failure:
basis (green) does not improve performance over the base ERM. Instead, the performance converges montonically to that of the base ERM model when m is full-rank.
In contrast to the behavior of the random basis, methods which utilize f. 9 or Lirain (table I exhibit a different structure with respect to While they all converge to the base ERM accuracy at full rank m (right end of the plot), at intermediate values of m we observe that projection improves the OoD performance of the network. More specifically, with increasing number of components, the performance first increases to an optimal choice of m and then decreases as more and more spurious nullspace components are added.
The best OoD performance is achieved by the optimized basis (red line fig. 3). which explicitly finds the directions leading to the highest marginal improvement in the training loss Cerain and discards directions which don't 't improve Ltrain. This suggests that the optimized basis finds a more meaningful set of directions for a given m compared utilizing PCA (that only considers g and not how f uses the different directions) or the Jacobian based feature spaces (which considers f's sensitivity to different directions but not whether the sensitivity contributes to improvement in the training loss Ltrain). Overall, this suggests that nullspace occupancy is mechanism of Ool failure, and that the optimized basis is the best choice of basis V with which to investigate this mechanism.
Does projection improve OoD performance? Next we study what improvements in performance are possible with perfect heuristics or "oracle" choices of layer and rank m on the DomainBed benchmark. In addition to ERM, we compare with the CORAL (Sun & Saenko (016), SagNet (Nam) et al. 2019), and aterdomain-Mixxu (Yan et al.) 2020 models (from DomainBed) which achieve tfeh--ff-the-ar performances the datasets explored in our work.
Heuristic for choosing m. As explained in (section 3) we report two oracles for each basis: 1) "oracle" assumes access to both the best layer well best number of components m, and 2) "oracle-layer which selects m using a heuristic and picks the best layer. Thus, "oracle-layer" a more practical indicator of achievable OoD improvements while "oracle" provides an upper bound on achievable performance. For the "oracle-layer" approach we choose m at the point when the training accuracy saturates to 99.9% of the full value.
Notice that for the optimized basis, the oracle numbers (o) compare favourably to the SagNet and CORAL algorithms on PACS, TerraIncognita as well DomainNet (suggesting possible improve- ments of 3.3 + 1.0 over SagNet on TerraIncognita, for example). Interestingly, the layer oracle (lo) which uses heuristic for choosing m also gives consistent improvements over ERM (table similar table with results on in-distribution generalization performance is included in Appendix
One concern when reporting oracle numbers as an aggregate, maximizing over m and choosing an oracle layer T., is if the reported improvements are significant or whether even a baseline model (at the same performance as the original ERM equipped with randomness in how it labels) could achieve the same improvements given X L trials. We compare against such baseline models (for the multi-class classification case) in Appendix C Our results show that the systematic improvements report in table are highly unlikely occur due to random chance (see Appendix C for details).
Does nullspace occupancy yield non-oracle improvements? We next study if it is possible to choose the layer/ and rank m using unseen training domain, to obtain performance improvements on a novel, held-out OoD test domain. Concretely, we train a network using standard DomainBed protocol on say d training domains, use unseen training domain d to pick and m, and
5

---

Published as a conference paper at ICLR 2023
Table 2: Aggregate domain generalization performance achieved by different bases RotatedMNIST, PACS, Terralncognita, and DomainNet. For each basis, we report the layer-oracle accuracy (lo) and the overall oracle performance (o). Error bars are in Appendix along with per-domain breakdowns
RotatedMNIST
PACS
Terralncognita
DomainNet
Imagenet-R
ERM
62.8
85.9
48.7
40.9
36.1
ERM Different Low-Rank bases Vn
RotatedMNIST
PACS
Terralncognita
DomainNet
Imagenet-R
lo
lo
lo
lo
lo
Random
62.8
63.2
85.6
86.4
48.7
49.3
40.7
40.9
36.1
36.1
Jacobian
62.9
63.7
86.6
87.1
49.3
50.6
41.0
41.6
37.2
38.2
PCA
63.0
64.3
86.9
87.4
50.1
51.6
41.4
42.2
37.1
37.4
Low Rank W1+1
62.8
63.4
85.9
86.7
48.7
50.0
41.1
41.6
Optimized
63.5
64.8
87.4
88.1
50.5
51.9
41.6
42.4
37.4
38.4
Representative state-of-the-art approaches:
CORAL
N/A
86.2 + 0.3
47.6 1.0
41.5 0.1
SagNet Mixup
N/A
86.3 + 0.2
48.6
= 1.0
40.3 + 0.1
N/A
84.6 + 0.6
47.9 = 0.8
39.2 = 0.1
Table 3: Aggregate domain generalization performance achieved by different bases RotatedMNIST; PACS, and Terralncognita using leave-one-out hyperparameter selection. We report mean results and standard errors over three independent trials.
RotatedMNIST
PACS
Terralncognita
ERM
65.6 + 0.7
78.2 + 0.8
39.3 + 1.1
ERM Different Low-Rank bases Vn
Random
64.6 + 1.3
76.4 + 1.6
34.8 + 2.3
Jacobian
65.4 + 0.7
78.1 + 0.8
38.1 + 2.4
PCA
65.7 = 0.6
78.1 + 1.3
38.9 + 1.3
Optimized
66.3 = 0.6
79.1 + 1.2
39.6 + 1.4
report performance after projection on test domain d + 2. Encouragingly, the optimized basis yields consistent improvements over ERM across all three datasets we evaluated on, namely, Rotated- MNIST, PACS, and Terralncognita (table 3 Further, the optimized basis is the only method that achieves consistent improvements over ERM in this setting. This demonstrates the value of learning the optimized basis that makes use of f. g and L (table over other baseline approaches for choosing the basis. Importantly, this also indicates that reducing nullspace occupancy is a viable method for improving OoD performance of already trained ERM networks in more practical, non-oracle settings.
nullspace occupancy distinct from "capacity control"? We compare against keeping the same activations 2 in layer l at test time, but instead reduce the rank of the matrix W4+1 in the next layer. This implements a simple, inference-time way to control the capacity of the function locally around layer l. Across all datasets, for both oracle as well as layer oracles table 2 we find that this yields inferior results to both the optimized basis as well as the PCA basis. This result demonstrates that it important take both the variance in the training data, well the overall behavior of into account when reducing the rank, as opposed to reducing the rank only based on local characteristics of the network in the next layer.
Is projection complementary to L2 regularization? L2 regularization (Krogh & Hertz 1991) is a commonly used heuristic for training deep models and is thought of commonly as regularizer. Given parameters 0 and loss function Ltrain. 1.2 regularization optimizes Ltrain + A |||0| where ||- denotes the L norm. is intuitive that L2 regularization should lead to "simpler" networks (in the linear case this corresponds to lower-rank solutions) which may not suffer from the nullspace failure mode. While the results table 2 already include models trained with weight decay (based on standard DomainBed hyperparameter sweeps), in this section further scrutinize the connection between nullspace occupancy failure and L2 regularization, since it is possible that in the presence of such regularization might extrapolate more smoothly in low-rank nullspace (see appendix F

---

Published as a conference paper at ICLR 2023
Mean 000 acc (per envionment
Accrrey relloo oweeeenoe encpeireed
ERM Optimized Basis (aracle)
train data accuracy
tast data accuracy (optimized)
0.6
test data accuracy (PC)
0.64
text data accuracy (jacobian]
0.4
test data accuracy (random)
explained
test data veriance explained
0.2
0.50
1e-5
1e-4
1e-3
1e-2
1e-1
0
20 30
40
50
Penaliy (A]
Number components (m)
Figure 4: (left) Oracle improvements (0) vs L2 penalty: For Rotated-MNIST, for each bin of L.2 penalty values and test domain, we train 15 networks and select one that achieves highest validation accuracy. We plot the OoD accuracy of selected networks averaged across test domains along with the average oracle improvement on the selected networks. We observe our method give comparable performance for all L2 penalty values. (right) Nullspace occupancy for poisoned test data: Classification accuracy after projecting network activations to the top-m components in the PC basis and random basis. The black and red curves the amount of total variance explained by the first-m components. We observe that for PC basis, test set accuracy increases till training variance explained is saturated and sharply declines when including nullspace components.
Instead of the standard DomainBed procedure, where for each set of train and test environments (termed condition) trains large number of ERM models via random hyperparameter selec- tion (Gulrajani Lopez-Pate Paz 2020) and picks the best model, we instead bin the networks into sets based on the strength À of the 1.2 penalty used train them (fig. 7 and then pick the best ERM model (using validation data from the training domain). This allows us to assess the complementary benefits of our approach (which completely projects down to a chosen subspace) over L2 regularization which should in principle ensure more smooth functions.
We plot performance of the best model in each L2 penalty bin and compare to the oracle im- provements achievable over that specific model using projection into the optimized basis (fig. 7 on RotatedMNIST (see Appendix B for more datasets). We find that projection gives consistent improvements over L2 regularization across the different buckets (x-axis fig. 7 right). This suggests that 1.2 regularization and projection are quite complementary to each other, and that nullspace failure occurs in deep networks even in the presence of 1.2 regularization.
Does nullspace occupancy emerge data poisoning? So far we studied a setup where we intervened by projecting the representation down to subspace Vn. and we observed that this can improve the performance instead did the reverse? That is, does the nullspace occupancy phenomenon emerge in cases where we explicitly force neural networks to not generalize?
the network f operating on the projected features v What we
To test this, utilize the experimental setting of (Huang et al. 2020 who show how take training
and test datasets Dtrain = {xi, y/: ) [Pirainl and Drest = (x; y/ )
Pieest and find a network up that generalizes poorly to Drest. Let X - P())) be the neural network mapping inputs X' to the distribution over the labels P()). Then we train to optimize:
Denain
Diaae
min Derran
llog uo (y:
log(1 uo (x))
|Deest
We perform our experiment on the MNIST dataset by splitting the dataset into train and test split. We essentially train perform well on the training set and badly on the test set, yielding small convolutional network up which by construction performs poorly on the given test set Drest. Having "poisoned"' the test set, we check projecting the test datapoints x E Prest to a subspace Vm can "de- "de-poison"" the network. fig. shows that this indeed happens in layer 5 for this network, and that one can recover a large fraction of the lost performance due to poisoning by projecting into the first components of the PCA basis that explain most of the variance.
This indicates that the phenomenon of nullspace occupancy contributing to generalization failure has applicability in other settings which not necessarily connected to out-of-distribution general-
7

---

Published as a conference paper at ICLR 2023
ization and hints at some interesting underlying mechanism that connects nullspace occupancy to generalization more broadly. See Appendix GG for additional experimental details.
Finally, performed experiments on adversarial robustness (see appendix K) where we found similar trends as data poisoning namely that both PCA as well as the optimized basis lead to gains in robustness, but the gap between PCA and optimized basis is not as large as observed in more traditional OoD generalization tasks (such as on DomainBed).
5 RELATED WORK
(OoD) Generalization. There has been considerable interest in the out-of- distribution generalization problem (Gulrajani & Lopez-Paz 2020 Arjovsky et al. 2019; (Sagawa et al. 2019 Li et al. 2017a) where the goal is to generalize the classifier J 9 to novel input domains D. A number of popular approaches have been pursued including causal invariance (Arjovsky et al.
2019), gradient starvation (Shah et al. [020 Pezeshki et al. 22200) risk extrapolation (Krueger et al.
2020), meta-lcarning (Li & Lij 2017) distributionally robust optimization (Sagawa et al. [2019] |iiha
et al. 2017) etc.. Among these, causal invariance and gradient starvation (Pezeshki et al. 2020)
(or simplicity bias (Shah et al., 2020)) are quite relevant to us, since they prescribe certain failure modes which might be causing poor out-of-distributtin accuracy. In contrast to work causal invariance, our work does not require an underlying "true" causal model or annotation of different environments (except for the purposes of leave-one-our domain evaluation). While gradient starvation is concerned with features V1 and V2 where both are potentially useful, our work is focused on removal of additional/extra features, rather than using all the relevant features. In this sense, gradient starvation and our work are quite complementary. Another relevant approach is that of last-layer feature reweighting. Rooenfeld al. (2022) and Kirichenko al.) (022) show that ERM already learns some generalizable features by showing that retraining the last layer using access to the target distribution or non-spuriously correlated reference distribution improves OOD performance. In contrast, we show that it is possible to drop extra nullspace features without access to target or reference distribution to improve network performance.
Nullspaces, PCA, and Representation Learning. Low-rank feature spaces (primarily using PCA) have been utilized for detecting adversarial examples (Li & Li 2017 Carlini Wagner| 2017) and for improving adversarial robustness (Sanyal et al.) 2018). In contrast to these works, we are focused on "semantic" OoD generalization as opposed to adversarial examples. Other recent work has noted that neural networks learn feature spaces which are essentially low rank (Huh et al. 2021). Our work makes use of that observation to study OoD generalization and how choices of bases affect generalization. In contrast to these works, which largely utilize PCA, our work considers the downstream network f as well to identify the directions that are most predictive of the label on the training set. In this sense, our work is like deep version of partial least squares (PL.) (Geladi & Kowalski 1986) which aims to find the discriminative directions in feature space for linear models.
Low-rank models and Pruning. There is a lot of work in model compression that focusses on reducing the rank of the layer weights W; after training (Cichocki et al., 2016 Udell et al. 2016). A naive data-independent strategy includes minimizing the distance [[ UVT between the original network layer weight W and the compressed weights UVT, using a closed form solution such as SVD DyDtonn et al. (2014); Novikov al. (2015). We utilize these techniques in our low-rank W++1 baseline (section 2 to compare against our feature projection techniques (table (2). In contrast to these works which focus on compression, our goal is instead to improve downstream Ool performance (our low-rank basis does not yeild any model compression). (Zhang et al., 2021) relates model pruning to -itto--disttibution generalization, showing that even in models which rely on spurious correlations, there exist subnetworks that are robust to spurious correlations. Similar to this work, we reduce the capacity of the network (or a particular layer in our case) to study OoD generalization. However, unlike network pruning, our technique focuses on particular layer and exposes nullspace occupancy as a mechanism by which OoD failure can happen.
6 DISCUSSION
Nullspace removal is not sufficient for improving generalization. We observe that while there usually exists a layer E (1,..., ,.) where performing the projection improves the oracle out of
8

---

Published as a conference paper at ICLR 2023
distribution accuracy, it is not necessary that the projection helps in every layer. To understand this, it is helpful to think of a breakdown of error e into two different sources, namely Enall which is the nullspace error and Eds which is the distribution shift error. One have distribution shift even if the test data spans the top Vm components which explain the training data very well since even the components are same, the distribution of the test datapoints could end up being quite different. Thus, it is not necessary that removing the nullspace error Exull would lead to a low overall error e. However, in practice it seems play role in OoD generalization.
Nullspace occupancy is not necessary for poor generalization. It is also straightforward to notice that, if trained network learns appropriately smooth functions f(v1, V2), ignoring an extra feature V3 = 0 for example, then it would not be troubled by test data occupying the nullspace. In practice, appears that networks learnt even in the presence of weight decay do not show such behavior which suggests that nullspace occupancy might be a mechanism for explaining generalization failure in those cases.
Connection to other OoD generalization failure modes. Given input features {Vi, v2} V present in a dataset Dtrains and a trained neural network function f(vi, V2) from V - y, where y is the label space (fig- D. recent work has studied two popular modes of out-of-distribution generalization failure with deep learning:
Causality: Given label y, a causal feature v and "spurious" correlated feature v2, the idea is to learn a function (((11) that ignores V2 (Arjovsky et al. 2019). Instead, if we learn a function f(V2), we fail due to spurious correlation.
Gradient Starvation (Pezeshki et al., 2020; Shah et al. 2020): Given two predictive features Vi and V2, where vi elicits stronger response at initialization than v2, the dynamics of learning yields ((v1) ) that ignores the weaker, but predictive feature V2. This can lead to generalization failure a novel environment.
Our approach is distinct from causality in the sense that we are simply measuring geometric properties of the learnt features as opposed to assuming any underlying causal model that induces OoD failure. More closely related to our work is gradient starvation. In the language of gradient starvation, let us assume stronger feature V1. a less dominant feature V2 and an invariant feature V3 0. Gradient starvation considers the case of V1 and v2, and suggests that the strong feature V1 might inhibit the learning of the weak feature V2, at least in the neural tangent kernel regime (Jacot et al., 2018). A priori, one might expect that V1 would suppress the learning of V3 as well, but our work shows empirically that this is not the case. Our initial attempts to extend the theory of (Pezeshki et al. 2020) revealed that their framework cannot easily accommodate notion of a "useless" feature, and the linearity assumptions in the NTK regime might not explain the non-smooth behavior of deep neural networks (see Appendix D). However, it is interesting to note that while the gradient starvation work attempts to mitigate this and learn more features, our work in contrast attempts to discard uninformative and redundant features.
7 CONCLUSION
In this work we introduced the concept of nullspace occupancy - namely when test data occupies the nullspace of a low-rank feature space Vn, that captures the training set variability and connected it to OoD generalization. We showed that with a careful choice of the basis Vns. systematic improvements in OoD generalization can often be obtained by projecting out the remaining nullspace components for test data points. We found this "optimized basis" by performing an optimization over orthogonal matrices to identify a set of components that minimizes the area under the training loss curve for different choices of rank m. On domain generalization experiments on RotatedMNiST. PACS and Terralncognita we found that this choice of basis has the potential to improve out-of-distribution accuracy of ERMs. Finally, we also found that nullspace occupancy emerges in a setting where one poisons the network to perform poorly on a predetermined test set. Together, these results hint at a broader interplay between nullspace occupancy and generalization in deep learning. There is much to explore to understand the ubiquity of this novel mode of generalization failure and further refine our approach to mitigate it, and hope the community works to address these issues in the future.

---

Published as a conference paper at ICLR 2023
REFERENCES
Ehab A AlBadawy, Ashirbani Saha, and Maciej A Mazurowski. Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing. Medical physics, 15(33:1150-115
2018.
Michael A. Alcorn, Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) pose: Neural networks are easily fooled by strange poses of familiar objects. 2018. doi: 10.4850/ARXI..I111.
(nttts:/://are
wttgg/bbb//11111 11553
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. July 2019.
Sara Beery, Grant van Horn, and Pictro Perona. Recognition in terra incognita. July 2018.
Nicholas Carlini and David Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, pp. 3 14. Association for Computing Machinery, New York, NY, USA, November 2017.
Andrzej Cichocki, Namgil Lee, Ivan Oseledets, Anh-Huy Phan, Qibin Zhao, and Danilo P Mandic. Tensor networks for dimensionality reduction and large-scale optimization: Part 1 low-rank tensor decompositions. Foundations and Trends@ in Machine Learning. 9(4-5):249-429, 2016.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020.
Dengxin Dai and Luc Van Gool. Dark model adaptation: Semantic image segmentation from daytime to nighttime. 2018. doi: 1044550/ARXX..1810005557 URL 1810 02575
Emily Denton, Wojeiech Zaremba, Joan Bruna, Yann LeCun, and Rob Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. Advances in neural information processing systems, 27, 2014.
Tomer Galanti and Tomaso Poggio. Sgd noise and implicit low-rank bias in deep neural networks. 2022. doi: 10.48550/ARXIV.22066000
URL
05
Paul Geladi and Bruce R Kowalski. Partial least-squares regression: a tutorial. Anal. Chim. Acta, 185:1-17, January 1986.
Muhammad Ghifary, W Bastiaan Kleijn, Mengjie Zhang, and David Balduzzi. Domain generalization for object recognition with multi-task autoencoders. August 2015.
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A Mann. Improving robustness using generated data. Advances in Neural Information Processing Systems, 11:4218-42333 2021.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. July 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song. Jacob Steinhardt, and Justin Gilmer, The many faces of robustness: A critical analysis of --i-riiiittitnoionn generalization. In 202/ IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, October 2021.
W Ronny Huang, Zeyad Emam, Micah Goldblum, Liam Fowl, Justin K Terry, Furong Huang, and Tom Goldstein. Understanding generalization through visualizations. In Jessica Zosa Forde, Francisco Ruiz, Melanie F Pradier, and Aaron Schein (eds.), Proceedings on "I Can't I Believe It's Not Better!" at NeurIPS Workshops, volume 137 of Proceedings of Machine Learning Research, pp. 87-97. PMLR, December 2020.
Minyoung Huh, Hossein Mobahi, Richard Zhang, Brian Cheung, Pulkit Agrawal, and Phillip Isola. The low-rank simplicity bias in deep networks. arXiv preprint arXiv:2103. 10427, 2021.
10

---

Published as a conference paper at ICLR 2023
Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. June 2018.
Bobak Kiani, Randall Balestriero, Yann Lecun, and Seth Lloyd. projUNN: efficient method for training deep networks with unitary matrices. March 2022.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Last layer re- re-training is sufficient robustness to spurious correlations. arXiv preprint arXiv:2204.02937, 2022.
Tamara G Kolda and Brett W Bader. Tensor decompositions and applications. SIAM review, 51(3): 455-500, 2009.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. In J Moody, Hanson, and Lippmann (eds.), Advances in Neural Information Processing Systems, volume 4. Morgan-Kaufmann Kaufmann, 1991.
Advancess
David Krueger, Ethan Caballero, Joern- Henrik Jacobsen, Amy Zhang, Jonathan Binas, Remi Le Priol, and Aaron Courville. Out-of-Distribution generalization via risk extrapolation (REx). March 2020.
Da Li, Yongxin Yang, Yi-Zhe Song, and Timothy M Hospedales. Learning to generalize: Meta- Learning for domain generalization. October 2017a.
Da Li, Yongxin Yang. Yi-Zhe Song, and Timothy M Hospedales. Deeper, broader and artier domain generalization. October 2017b.
Xin Li and Fuxin Li. Adversarial examples detection in deep networks with convolutional filter statistics. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5775-5783, October 2017.
Gregory I. Murphy. The big book of concepts. 555, 2002.
Hyeonseob Nam, Hyunjae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing domain gap by reducing style bias. October 2019.
Alexander Novikov, Dmitrii Podoprikhin, Anton Osokin, and Dmitry P Vetrov. Tensorizing neural networks. Advances in neural information processing systems, 28, 2015.
Francesco Orabona and Dávid Pál. Optimal non-asymptotic lower bound on the minimax regret of learning with expert advice. arXiv preprint arXiv: 1:Xxx:55/1.02/700 2015.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for Multi- domain adaptation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, October 2019.
Mohammad Pezeshki, Sékou-Oumal Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoic. Gradient starvation: learning proclivity in neural networks. November 2020.
Elan Rosenfeld, Pradeep Ravikumar, and Andrej Risteski. Domain-adjusted regression or: Erm may already learn features sufficient for out-of-distributtin generalization. arXiv preprint wrXiv:2202.068566 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for Worst-Case generalization. November 2019.
Amartya Sanyal, Varun Kanade, Philip H S Torr, and Puncet K Dokania, Robustness via deep Low-Rank representations. April 2018.
Harshay Shah, Kaustav Tamuly, Aditi Raghunathan, Prateek Jain, and Praneeth Netrapalli. The pitfalls of simplicity bias in neural networks. In NeurIPS. June 2020.
Aman Sinha, Hongseok Namkoong, Riccardo Volpi, and John Duchi. Certifying some distributional robustness with principled adversarial training. October 2017.
11

---

Published as a conference paper at ICLR 2023
Baochen Sun and Kate Saenko. Deep CORAL: Correlation alignment for deep domain adaptation. July 2016.
Madeleine Udell, Corinne Horn, Reza Zadeh, Stephen Boyd, et al. Generalized low rank models. Foundations and Trends@ in Machine Learning, 91):1-111, 2016.
N Vapnik. An overview of statistical learning theory. IEEE Trans. Neural Netw, 10(5):988 September 1999.
Shen Yan, Huan Song, Nanxiang Li. Lincan Zou, and Liu Ren. Improve unsupervised domain adaptation with mixup training. arXiv preprint arXiv:2001.0067 2020.
Dinghuai Zhang, Kartik Ahuja, Yilun Xu, Yisen Wang, and Aaron Courville. Can subnetwork structure be the key to Dut-of-Distribution generalization? June 2021.
12

---

Published as a conference paper at ICLR 2023
We include the following appendices:
We report per-domain domain generalization results for projection in all bases on SorraeddMNNSSS PACS, Terralncognita, and DomainNet.
Appendix A
Appendix We provide figures from the main text additional datasets.
Appendix We demonstrate that our reported oracle results are not artifact of considering the maximum of many models. To establish this, we compare achieved improvements against the maximum performance achieved by large of models with the same accuracy the base ERM, equipped with randomness.
Appendix We discuss the connection between nullspace components and the theoretical framework of Gradient Starvation (Pezeshki et al. 2020).
Appendix B We include a formal discussion of weight decay and how it relates to the
nullspace phenomenon
We report results for projection in-distribution generalization for Rotated- MNIST, PACS, Terralncognita, and DomainNet.
Appendix F
Appendix We report additional details for the data poisoning experiment.
Appendix We interpolate between test datapoints (with components the nullspace) to the projected test datapoints 2mh and study how accuracy varies as we move between them
Appendix T We present the small convolutional architecture used for all experiments on MNIST.
Appendix D We include plots of cut-of-distributiin accuracies after projection into all the different bases, for all networks we consider across every dataset.
Appendix K
We study the extent to which the optimized basis can help defend against adversarial attacks.
Appendix We include pseudocode for computing the oracle and layer-oracle accuracies reported the paper.
L
Appendix M
We include pseudocode for obtaining optimized basis by optimizing
equation
A PER-DOMAIN DOMAIN GENERALIZATION RESULTS
In this section, we show per-domain results of Table 2 from the main text. For each test domain, we report the mean accuracy and SEM over three nets selected using DomainBed search protocols, after projection into different low rank feature spaces.
Table 4: Per-domain oracle (o) and layer-oracle (lo) out-of- classification accuracies for different choices of bases -MNaTe--MNIST..
Algorithm
15
30
45
60
75
Avg
ERM
100.0 + 0.0
96.4
0.0
79.1
1.1
49.9
1.8
31.6 +
19.9
1.0
62.8
Random (o)
100.0 + 0.0
96.5 +
79.7 + 1.6
50.4 + 1.6
32.1 + 1.5
20.7 + 2.1
63.2
Jacobian (o)
100.0 + 0.0
96.7 0.1
80.2 + 1.4
50.8 + 2.1
32.3 = 1.9
22.0 - 0.7
63.7
PCA (o)
100.0 + 0.1
9.1
80.9 + 1.2
31.611.
34.6 = 1.8
21.6
0.5
64.3
Low Rank Wi+1 (o)
100.0 + 0.0
9.0
79.6
1.1
50.3 1.8
32.6 = 1.5
21.4
63.4
Optimized (o)
100.0 + 0.0
9.1
81.0
1.2
52.4
36.1
1.6
22.0 + 1.0
64.8
Random (lo)
100.0 + 0.0
9.04
79.1
+ 1.1
49.9 + 1.8
31.6 + 2.0
20.0 + 1.0
62.8
Jacobian (lo)
99.9
96.4
79.3
1.4
+..0
31.8
2 2.1
20.8 + 1.2
62.9
PCA (lo)
0.0
96.40.11
79.2 1.0
50.7
1.2
31.7 + 1.7
20.4 1 0.8
63.0
Low Rank Wi+1 (lo)
100.0.0000
96.5
7.10
4..9
31.31.6
19.8 + 1.0
63.4
Optimized (lo)
99.9
96.4
79.7
4.2
35.2+1.
20.3 + 1.0
63.5
13

---

Published as a conference paper at ICLR 2023
Table Per-domain oracle (o) and layer-oracle (lo) out-of-distribution classification accuracies for different choices of bases on PACS.
Algorithm
A
C
P
S
Avg
ERM
87.9 + 1.2
81.4 1.7
96.1 + 0.5
78.0 1.7
85.9
Random (o)
87.9 1.2
81.8 = 1.0
96.3
79.7 1.4
86.4
Jacobian (o)
88.6 # 1.2
83.6 1.6
9.64
79.7 = 1.6
87.1
PCA (o)
88.9 0.9
83.7 1.6
9.5
80.7 1.0
87.4
Low Rank Wi+1 (o)
88.5 + 0.7
83.2 + 1.6
9.51
79.0 = 2.2
86.7
Optimized (o)
89.1 1.0
84.2 1.8
96.7
82.4
0.3
88.1
Random (lo)
86.9 + 1.1
81.0 - 0.8
9.6
78.1
1.4
85.6
Jacobian (lo)
87.9
82.9 1.4
9.6
79.4 = 0.6
86.6
PCA (lo)
08.51.00
82.9 - 1.7
9.5
80.1 1.3
86.9
Low Rank Wi+1 (lo)
87.9 + 1.2
81.4
78.3 9.51
85.9
Optimized (lo)
88.0 + 0.9
82.9
= 1.9
9..5
82.0 = 0.5
87.4
Table Per-domain oracle (o) and layer-oracle (lo) out-of-distribution classification accuracies for different choices of bases Terralncognita.
Algorithm
L100
L38
L43
L46
Avg
ERM
53.6 + 3.5
43.9 + 1.2
56.8 + 1.5
40.4 + 1.2
48.7
Random (o)
54.5 + 3.7
44.7 + 1.8
57.4 + 1.9
40.7
0.4
49.3
Jacobian (o)
56.9 + 2.7
46.0 + 1.0
57.8 + 1.4
41.7 = 1.2
50.6
PCA (o)
58.1 2.1
47.6 0.8
58.7
41.9
1.7
51.6
Low Rank Wi+1 (o)
55,8 +
46.3 1.0
57.0
41.0 1.8
50.0
Optimized (o)
58.6 2.2
47.9 = 2.4
59.3 1.3
41.9 1.1
51.9
Random (lo)
53.5 0.3
44.3 + 1.3
2.1
39.9 0.8
48.7
Jacobian (lo)
55.1 3.1
44.2 = 1.4
57.2
41.0
49.3
PCA (lo)
2.0
45.4 0.9
57.0
41.1 1.3
50.1
Low Rank W4+1 (lo)
53.6 3.6
44.0 + 1.2
56.8 1.5
40.4 1.2
48.7
Optimized (lo)
56.1 + 3.4
46.9 + 2.0
57.6 1.3
41.1 1.1
50.5
B ADDITIONAL PLOTS
In this section, we present figures from the main paper evaluated on additional datasets.
In Figure 3 of the main paper, show that nullspace accuracy relates to OOD failure by plotting the improvement in OOD classification accuracy against the dimensionality of the subspace used for many different bases for Rotated-MNIST PACS, and TerraIncognita. Fig shows the corresponding plot for one randomly selected network from DomainNet.
In Figure 4 (right) of the main paper, verify that the effect of projection is complementary to that of weight decay by plotting the improvements obtained by projection along with the base network accuracy for networks trained with different values of weight decay on -NISS. fig. presennss the corresponding result on PACS, where we again see that projection gives consistent improvements across different buckets of weight decay, suggesting that weight decay and projection are complementary to each other.
C COMPARISON WITH MAXIMUM PERFORMANCE OF A LARGE NUMBER OF BASELINE MODELS EQUIPPED WITH RANDOMNESS
In the main text, report oracle numbers that obtained by maximizing many choices of subspace dimensionality. and oracle layer These numbers are the maximum over many different trials, raising the concern that they may not be significant because even baseline model with the
14

---

Published as a conference paper at ICLR 2023
Table Per-domain oracle (o) and layer-oracle (lo) out-of-distrriution classification accuracies for
different choices of bases on DomainNet.
Algorithm
clip
info
paint
quick
real
sketch
Avg
HRM
59.6 + 0.8
19.9 0.3
45.5 + 0.9
12.2 + 0.3
59.2 + 0.6
49.2 + 0.7
40.9
Random (o)
59.6 + 0.8
19.9 0.3
45.5 + 0.9
12.2 - 0.3
59.2 0.6
49.2 + 0.7
40.9
Jacobian (o)
60.3 + 0.9
20.5
= 0.4
46.3 0.9
13.0
- 0.1
59.9 0.6
49.8 + 0.5
41.6
PCA (o)
61.0 0,9
21.0 0.4
4.94
13,8 = 0.1
60.3
50.8 +
42.2
Low Rank Wi+1 (o)
59.9 + 0.8
0.4
0,9
13.0
59.8
0.7
41.6
Optimized (o)
61.1 + 0.9
21.0 0.4
47.0
0.9
13.5
61.1 0.7
51.0 + 0.8
42.4
Random (lo)
59.4+ 0.7
1.2
45.5
0.2
+.2
49.2 0.7
40.7
Jacobian (lo)
59.7
2..0
45.5
45.5009 0.9
12.5 0.2
4..3 0.6
41.0
PCA (lo)
60.1 1.0
0.3
1.07
12.7
0.3
0.6
4.99
41.4
Low Rank W1+1 (lo)
59.4
20.1 10.1000 0.3
45. 0.9
12.5
1255022 0.2
5.4
4.
41.0
Optimized (lo)
60.7 0.9
21.0 + 0.4
45.8 1 1.1
12.9
= 0.3
59.6
4.87
41.6
Improvenent in 0OD accuracy
0.10
bass Optimiasd
Jacosten
9.05
Random
-2.05
-3.10
100
300
500
Number of components (layer 2)
Figure 5: Nullspace occupancy relates OOD failure: Projecting test OOD activations onto subspace of rank m (x-axis) improves OOD performance (y-axis). We randomly pick one of 18 networks trained on DomainNet and the layer which best explains the nullspace occupancy related failure mode.
same accuracy as ERM equipped with randomness labelling could achieve the same improvements given m x L trials. In this section, we construct one reasonable choice of such a model, and compare the improvements obtained via our oracle method with an upper bound of the improvement obtainable via the random baseline model.
Let the base (ERM) model be M with classification accuracy x on the OOD set. Let the number of samples in the test set be the number of layers I, and the number of considered choices of feature space dimensionality be m. In this setting, the effective number of experiments We therefore consider set of non-deterministic models W,...., Wa, where each model W. predicts label correctly for each test sample with probability x. Let the out-of-distribution accuracies of =(1=x)
m.
models Wi,... Wa be random variables Ti,..., Td. It follows that E [x:] æ, and [x:]]
We wish to compare the improvements obtained by our method with E [ma.::e]] To obtain an
upper bound for this expectation, we assume that variables I; ~ Nxx =(1-#l) We then use the following result:
Theorem Orabona & Pál (015)) Let XI,..., Xa be independent Gaussian Random Variables 02), For any d 2,
-vInd
))
( (222nd 2n +
2
2
/ logd d 2 E [matiela] 2 exp
o
6.35
This gives the upper bound:
x(1 2log(mL)
E [ma 2:] xx o x
n
15

---

Published as a conference paper at ICLR 2023
PAGS
Imp
0.00
0.00
16-6 16-5
10:4
L6-2
162
Weigh So0p
Figure 6: Oracle improvements (o) vs weight decay: For PACS, for each bin of weight decay values and test domain, we train 15 networks and select the one that achieves highest validation accuracy. We plot the OOD accuracy of selected networks averaged across test domains along with the average oracle improvement on the selected networks. We observe our method give comparable performance for all weight decay values.
We compare this upper bound with the oracle performance obtained via projection on the ImageNet-R dataset, which offers a tight upper bound due availability of a large number test samples. In our experiments on ImageNet-R, we consider projections in 5 layers, and m 32 choices of dimensionality of feature subspace. The base accuracy of ERM is = 36.1%. yielding an I 0.88%. In contrast, the oracle accuracies of our projection method yield performance improvements of 1.3% when using the PC basis, and 2.3% when using the optimized basis. These results suggest that our method provides a systematic improvement to generalization performance, and not an artifact of taking maximum over a large number of models.
upper bound of [ma
Note that the baseline model we consider, constructed by using the maximum performance with set of models that achieve the same accuracy as the base model, is a very strong baseline. For reference, we consider a naive model equipped with randomness that starts with the base ERM and randomly flips the prediction with probability p. it would achieve accuracy p + (1 p) #dasses This could be significantly lower than x and the maximum of large number of such models could still yield performance less than x.
D CONNECTION TO GRADIENT STARVATION (PEZESHKI ET AL.. 2020
Consider our running example of training set with 3 features {V1, V2, 1}. where feature Va a nullspace feature, and has a constant value of Gradient Starvation (GS) considers the setting where is "stronger" feature than V2, and suggests that the strong feature inhibits the learning of the weak feature V2. the context of our work, one might expect that vı would also inhibit the learning of V3. thus potentially mitigating generalization failure due to nullspace occupancy. However, our empirical results indicate that this is not the case.
To reconcile this discrepancy, attempt to fit nullspace features within the theoretical framework of Gradient Starvation. GS considers a setting with n data points = .x......%0 x,, E RTxm, labels y == f=, 13" and Y diag(y) R"x", and function output y(X) fXX for neural network f. GS looks at the NTK parameterization, in which the network output is approximated as linear model of the function of the parameters, near initial parameters 0,:
(XX 0) xXX, 0o) + .
(5)
where 4b = (X, On) is the neural tangent random feature (NTRF) matrix at initialization, i.e.: Φ 0) (X,0) example, where the response defined as the deviation from its initial value:
d0 E _ GS then seeks to characterize the response of the network to training
r Y ýo) Y
(6)
The theory studies features defined in terms of the singular value decomposition of the NTRF:
Y USVT
16

---

Published as a conference paper at ICLR 2023
where (VT),. is the jth feature, (S)zz is the singular value of that feature and (U),, are the weights of the feature in each example adjusted according to the labels y. The network's 's response to each feature can then be expressed as
z -
(8)
To evaluate the prediction of the Gradient Starvation framework to nullspace features, our goal is to look at the response to nullspace features, when they appear in new out-of-disrribution example tee Note first that features considered under the framework are obtained via singular value decomposition of the NTRF, and the diagonal matrix of singular values S has rank n, i.c. all the singular values are positive. Hence, the Gradient starvation framework does not include any features with singular value zero (nullspace features), and can only explain suppression of features in the training subspace.
To attempt to extend the framework to include nullspace features, extend the feature set obtained via SVD of the NTRF, by extending the set of examples [x1,..., Xn. x.) [x++))xm to now also include sample To. a out-of-distribution example that contains variation along some nullspace feature Vo. Looking at the response
SV we note that S would now have an additional element, leading to a nonzero singular value along V.. However, evaluation training dynamics in GS looks at optimizing 0 in the assumed linear approximation from equation 5 via minimizing regge-regulrrzzed cross entropy loss, and any amount of regg-rreularizationn would make it such that the component of 0 along Vo would go down to exactly zero. As result, the response feature j would have nonzero corresponding values in S and VT, but a coefficient of O in 0. Therefore, this extension of the GS framework would predict a network response of zero along any nullspace component, at odds with the empirical results we find in this work.
SV 0 along nullspace
E WEIGHT DECAY AND PROJECTION: AN ANALYSIS
Let us consider the weight decay regularization and how that relates to the eigenvalues of the weight matrices Wi E Rm K learnt in different layers 1. Specifically, weight decay minimizes the Frobenius norm W || F. which can also be written as tr(W- WT). Consider the Singular Value Decomposition (SVD) of W UEVT, rank for W, and let u, refer to a column of U. We can then write:
tr(W- WT) Ir(USV"VEU")
(r(UEPUT)
(10)
tr( Σ ofu, uz
(11)
oftrl -
(12)
Σ i=1
of
(13)
Where in the first line we used the fact that VTV 1, then used the fact that tr (trace) is linear operation, and then used the fact that ||w:]| = 1.
Thus, adding weight decay to a network yields small singular values (in an L-2 sense). This, however does not yield sparsity. Thus, in some sense, "full-rank" information can potentially pass through multiple layers of a network with weight decay (this would not be possible if the rank r was really small (or o; = 0) in contrast, for example.
This explains the complementary benefits of performing projection even in the presence of weight decay (as shown in the main paper).
F IN-DISTRIBUTION GENERALIZATION RESULTS
In this section, we report the performance of layer-oracle and oracle projection methods on in- distribution generalization in table table g
17

---

Published as a conference paper at ICLR 2023
Interestingly, find that the optimized basis consistently outperforms the other choices of bases. For example, on PACS, the optimized basis yields improvements of 0.9% as opposed to 0.3% for say, the PCA basis. Comparing the IID to the OOD setting, find that the PCA basis yields much smaller improvements compared to the optimized basis. The ratio of the improvements in oracle from PCA basis to the optimized basis (averaged across all datasets) is 0.42 for IID compared to 0.79 for OOD. This suggests that, consistent with our intuition, the OOD datasets have more features occupying the nullspace the PC basis and thus the PC basis performs better in OOD and worse in IID. However, considering the impact of the downstream function in addition to the variance (which our optimized basis does) can improve performance IID setting as well.
G DATA-POISONING EXPERIMENT DETAILS
We perform our data poisoning experiments on the MNIST dataset. We split the MNIST dataset into two halves: clean training set and a poisoned test set. We train a network with 4 convolutional layers, an average pooling layer, and linear layers to perform well on the training set and poorly on the test set. All convolutional layers have kernel size of 3 x 3, padding of 1, and 64, 128, 128, and 128 channels respectively. The second convolutional layer uses a stride 2, while all others use stride of 1. Each convolution is followed by ReLU nonlinearity, and GroupNorm operation with groups. After the convolutional layers, an average pooling layer with a kernel size x and a stride of is applied such that only single value remains per channel. The resulting representation is fed to 3 linear layers of sizes 128 x 64, 64 x 32, and 32 x 10 respectively. Each linear layer save for the last is followed by ReLU nonlinearity, and a batch normalization layer. To train the network, we use the Adam optimizer with 81 0.9, and B3 = 0.999. We train the network for 100 epochs with batch size of 64 and learning rate of 1e-3.
H HOW DO IMPROVEMENTS IN ACCURACY RELATE TO THE DISTANCE FROM THE SUBSPACE?
Do the observed improvements occur because we project to the subspace or is some other perturbation of the test data point x equally beneficial? To answer this, we consider an experiment where we partially project down the features V Formally speaking, each v can be written feature vectors
sum of two vf + vž such that vf lies in subspace Vm and v lies subspace V/7 For A E [0, 1]. we consider the partial projection to Vm as v v + Av If x 1, we recover the original feature vector (no projection) whereas = yields the full projection Vm (methods discussed above).
fig. shows the interpolation between X and the fraction of OOD improvement. contains the plots for this experiment across the 12 best ERMs PACS and Terralncogniaa. Overall, find that project out the component feature vector in V%, and thus increase the distance from the subspace Vn performance gradually decreases (roughly monotonically) for both the optimized basis well
Table 8: Aggregate in-distribution generalization performance achieved by different bases on Ro- MNtT, PACS, Terralncognita, and DomainNet. For each basis, we report the layer-oracle accuracy (lo) and the overall oracle performance (o).
RotatedMNIST
PACS
Terralncognita
DomainNet
ERM
99.2
96.6
91.5
59.3
ERM + Different Low-Rank bases Vm:
RotatedMNIST
PACS
TerraIncognita
DomainNee
0
lo
0
lo
0
lo
0
Random
99.2
99.2
96.7
96.8
91.6
91.7
59.4
59.4
Jacobian
99.2
99.3
96.6
97.0
91.9
92.2
59.6
60.3
PCA
99.1
99.3
96.8
97.0
91.8
92.2
59.7
60.4
Low Rank W4+1
99.2
99.4
96.7
96.9
91.5
91.6
59.7
60.4
Optimized
99.1
99.4
96.9
97.5
92.6
93.3
60.8
62.2
18

---

Published as a conference paper at ICLR 2023
as the PCA basis. This suggests that improvements are likely due the projection of the datapoints to the subspace Vm as opposed to some other kind of structured perturbation.
Fraction of GOo improvementt attained
1.0
Fraction of 00D improvement attained
Optimized basis
Optimized basis
0.8
0.8
0.6
0.6
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6
0.8
100
0.0
0.2
0.4
0.6
0.8
10
Fraction of distance between optimal subspace and point
Fraction of distance between optimal subspace and point
Figure 7: Fraction improvement vs distance from optimal subspace: For the optimal subspace Vm (i.e. optimal layer and m), project out À fraction of the component orthogonal to Vn. For the resulting feature vector, we plot accuracy improvement divided by the optimal subspace accuracy improvement (y-axis) fraction of distance from the optimal subspace, À (x-axis). Solid lines show the mean fraction improvement over 12 DomainBed ERMs on PACS (left) and Terralncognita (right), and shaded portions indicate the standard deviation.
I ARCHITECTURE FOR ROTATED-MNIST EXPERIMENTS
Following DomainBed, use the ResNet-50 architecture for all experiments PACS, TerraIncog- nita, and DomainNet, and a small CNN for experiments on Rotated-MNIST. The architecture for the small CNN consists of convolutional layers, succeeded by global average pooling and one linear layer. The convolutional layers are of kernel size 3x3 and "same" padding, with 64, 128. 128, and 128 channels each. Each convolution layer followed by a ReLU activation and GroupNorm layer with 8 groups
J ADDITIONAL PLOTS
In this section, we include plots of cuttof-diisributtio accuracies after projection into all the different bases, for all networks, across Rotated-MNIST (fig.8). PACS (fig. (9). TerraIncognita (fig. [o). and DomainBed (fig. IT). Following DomainBed methodology, we have three selected networks for each OOD domain (one from each independent trial), and plot the -utto--distriiution accuracies after projection into different bases in the oracle layer of each network.
19

---

Published as a conference paper at ICLR 2023
0.20
PC badss
0.20
Optim and basa
0.55
Jacob un basa
Pundom badia
6.05
0.05
-5.05
-3.05
-0.20
-0.30
-0.30
0.20
0.20
0.55
0.05
0.85
-0.05
-0.05
-0.05
0..0
0.30
-0.00
0.30
5.10
0.10
0.25
0.55
-0.05
-0.05
-9.05
0.30
-0.00
0.30
5:10
0.10
-0.05
-0.05
-9.05
-0.10
-0.30
120
120
0.30
5.30
0.10
-0.25
-0.05
-0.05
-0.30
-0.30
-0.30
82
120
120
220
0.20
0.30
0.30
0.50
0.50
-025
-0.05
-0.05
30
220
Figure 8: Improvements in out-of-distribation classification relative to full network Rotated- MNIST. The x-axis is rank subspace used for projection, while the y-axis the improvement in OOD classification accuracy relative to the base network. Each row of plots corresponds to different out-of-distributin test domain.
20

---

Published as a conference paper at ICLR 2023
bade Optimazed bads
0.25
Jocab on bass
Raadom bads
0.05
9.25
0.29
5.00
0.50
-0.05
30
100
130
200
230
35
100
130
220
250
20
30
50
60
0.30
6.10
0.30
0.50
0.50
-0.25
-0.05
-0.05
-0.30
-0.10
-0.30
30
0.20
0.20
0.55
-0.10
-0.10
-9.30
0.25
0.25
0.20
5.00
0.50
100
200
300
402
5.00
200
200
330
420
503
50
100
150
200
250
Figure 9: Improvements in out-of-distribution classification relative to full network on PACS. The x-axis is the rank of the subspace used for projection, while the y-axis the improvement in OOD classification accuracy relative to the base network. Each row of plots corresponds to a different out-o--distribution domain.
21

---

Published as a conference paper at ICLR 2023
0.19
PC basss
0.10
9.20
Optimezed bass
0.25
Jacoo an
5.05
0.55
-0.10
-6.10
-9.30
0.25
0.20
0.29
5.00
0.50
-0.05
200
400
500
500
1000
102
200
300
400
503
200
400
600
800
2000
0.30
0.30
0.50
0.50
-0.25
-0.05
-0.25
-0.30
-0.10
150
202
820
1020
400
1000
0.10
0.20
0.59
8.05
0.05
-6.05
-0.10
-5.10
-9.10
Figure 10: Improvements out-of-distribution classification relative full network on Ter- ralncognita. The x-axis is the rank of the subspace used for projection, while the y-axis is the improvement in OOD classification accuracy relative to the base network. Each row of plots corre- sponds to different out-of distrr--iitiittiion test domain.
22

---

Published as a conference paper at ICLR 2023
Ransto favts
0.20
5.00
0.50
-0.0
100
220
300
402
5.0
202
400
630
820
1009
300
1000
1500
2000
0.20
5.00
0.50
-0.05
-0.10
-4.10
200
4.0
500
500
1002
50
100
150
220
250
20
20
50
60
0.25
0.25
0.20
5.00
0.50
00.55
-0.10
-3.10
530
1000
1500
2000
1000
2000
100
250
0.25
0.25
0.00
0.00
0.00
-5.10
-3.10
300
100
400
250
0.25
0.25
0.50
0.00
0.50
-0.10
-0.10
-0.30
500
100
430
3020
1500
2000
0.25
5.05
0.55
-0.10
-0.10
-0.10
Figure 11: Improvements in out-of-distribution classification relative to full network on Domain- Net. The x-axis is the rank of the subspace used for projection, while the y-axis is the improvement OOD classification accuracy relative to the base network. Each row of plots corresponds to different
test domain.
K ROBUSTNESS TO ADVERSARIAL PERTURBATIONS
In the main paper, showed how alleviating nullspace failure leads to improved OOD generalization. Can we also improve performance on adversarial examples generated to fool the full network by projecting limited number of components? To check this, we perform experiment where
23

---

Published as a conference paper at ICLR 2023
evaluate oracle performance after projection adversarial examples generated using AutoAttack (Croce & Hein 2020). for a WideResNet-28- 10 network trained on CIFAR-10. We consider attacks with maximum Linf norm of 8/255. We report oracle accuracies for the PC and optimized bases, as well as corresponding CIFAR-I R-1AR110 10 test set performance in table | and plot accuracies for all choices of number of components in fig. [2] We additionally include in table results for the augmentation driven adversarial defense of Gowal et al. (202I). a representative state-of-the-ar method for this dataset and architecture. We find that both the PC basis and optimized basis recover meaningful amount of the network's 's performance on the adversarial set.
Test set
Adversarial set
ERM
94.29%
0%
PC basis
78.52%
46.48%
Optimized basis
84.77%
44.92%
Gowal et al.
2021
87%
62%
Table Oracle performance after projection on the adversarial set, along with corresponding clean test set accuracies on CIFAR-IO.
Teit set bado
Adverarial IFC
30
14
Number components Ony
Figure 12: Nullspace occupancy for adversarial examples. Classification accuracy on CIFAR-10 test set and adversarial example set after projecting to the top-m components of the PC and optimized bases in layer 0, the oracle layer.
24

---

Published as a conference paper at ICLR 2023
L PSEUDOCODE FOR COMPUTING ORACLE AND LAYER-ORACLE ACCURACIES
In this section we present pseudocode for the procedure used to report oracle and layer-oracle accuracies given the OoD and IID accuracies of network for all choices of layer and number of components.
Algorithm! Pseudocode for computing oracle and layer-oracle accuracies for one network given OoD and IID accuracies for all considered numbers of components and layers
1: Input: Layers L = {O....,4 1}, OoD accuracy matrix A R IID accuracy matrix
B IR
2: Set oracle layer 1, f- arg m (O.... Aú m-1 1) such that B 1l 2 0.999 x B B m - 1] A [o.j]
3: Set heuristic index +- [o..,m
4: Oracle accuracy t-
5: Layer-oracle accuracy lo f- Allo, k]
M PSEUDOCODE FOR OPTIMIZING UNITARY MATRICES TO OBTAIN THE OPTIMIZED BASIS
In this section we present pseudocode for optimizing the objective equation 3 using the unitary optimization procedure of Kiani et al. (2022
Algorithm 2 Pseudocode for obtaining the optimized basis
1: Input: Encoder to target layer, classifer f from target layer, training mini- batchesthess ((x;, y.) } training loss Ltrains step size a, PC basis Vpex Hpc
2: Initialize basis < Vpes P f- Hpc
3: for E {1, (1,...,NV N) do
4: Set loss 1 - 0
5: Get representation for target layer z g(x:)
6: for number of components k E {1.... m| do
7:
Project representation using eq [ to get using z, V, and components. Classify projected representation < f(2)
8:
9:
I + I Ltrain (ú. 9)
10:
end for
Grad g Vvl
12: Project gradient <- 0.5 (g Vg
13: Update basis V + exp (-aVTp) p)
14:
end for
15:
Return V,μ
25


Reviews:
[{"review_id": "yv65hsRLa9", "reviewer": "Reviewer_k2wc", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this work, authors hypothesize that a failure mode of neural networks corresponds to the case where models depend on features in the null-space of the training risk. That is, representations learned through ERM preserve non-negligible variance along directions that do not affect the training risk itself. Authors then claim that to be a source of generalization gap since train/test variations across said directions could affect the risk estimated on unseen data. To counter that, a scheme is proposed to learn extra parameters to compress representations and fine tune part of a network. Experiments are carried out under the domain generalization setting where improvements over vanilla ERM are observed.", "summary_of_the_review": "The proposal is novel and interesting, and discusses properties of representations learned with very popular approaches. My main concerns lie in the fact that it's unclear how the proposal would induce better representations for domain generalization. While authors did report clear evidence of improving performance, it's unclear why that is.", "strength_and_weaknesses": "Pros:\n\n+The paper reveals interesting issues in representation learning: representations preserve variance along directions that are irrelevant to the training risk.\n\n+A simple and somewhat practical approach to compressing representations is proposed and, while authors focused on cases where training and testing distributions differ, it could also be useful in more standard i.i.d. cases.\n\n+Under domain shifts, a large set of experiments shows consistent improvements over vanilla ERM, known to be a strong baseline for domain generalization benchmarks.\n\n\nCons:\n\n-It is unclear to me how compressing representations would affect generalization under distribution shifts. To be clear, it seems to me that compressing away the risk null-space will affect generalization in the i.i.d. setting. As one would expect, focusing on the subset of risk minimizers that most compress data enables better generalization. However, why should we expect that to be helpful under distribution shifts? For instance, a widely discussed kind of shift known to affect performance is the existence of spurious features that correlate with labels only in the training sample (e.g., cow on grass vs. cow on the beach). Such spurious features would very likely be preserved after compression since they are so effective in minimizing the risk estimated with training data.\n\n-Also related with the point made on the above, it's unclear which domain generalization setting is considered or which kinds of assumptions are expected to hold in order for the proposal to work. There are several domain generalization settings in the literature, each associated with a set of assumptions over the relationships between training and testing data sources, and different methods are better suited for different settings. It seems to me that the proposal focuses on the covariate shift scenario where data marginals shift, but not class conditional distributions. Is that so? And then again, why should we expect compressed representations to work better than their non-compressed counterparts in this scenario? It could be the case that compressing introduces domain invariance, but that should be verified. I would suggest measuring the H-divergence [1] or some other distance between train and test distributions with and without compression.\n\n-It could be the case that the proposal, rather than offering some mechanism to enable learning from different domains, behaves as a regularizer in the standard sense: the projection step projects a model onto a \"simpler\" subset of risk minimizers, with lower-rank representations. If so, we should observe similar performance improvements on standard i.i.d. cases. Could it be the case that observed improvements over ERM are simply due to obtaining a better in-domain classifier?\n\n-Minor comment: while in the text it is stated that the proposal does not require domain annotations, those are necessary to enable scheme such as leaving a training domain out for deciding on projection (hyper)parameters.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "yU9jVgp6HYg", "reviewer": "Reviewer_tTTa", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work identifies a particular failure mode of OoD generalization for discriminative classifiers and introduces the concept of nullspace occupancy when test data occupies the nullspace of low-rank feature space and captures the training set variability. This work finds that with a careful choice of the basis V_m, the out-of-distribution accuracy often improves by projecting out the remaining nullspace components for test data points.", "summary_of_the_review": "This is a paper tackling an important problem. The observation is interesting and the writing is clear but these are some detailed issues that need to be clarified.", "strength_and_weaknesses": "Strengths:\n\nS1. The general problem of OOD detection is an important problem in reliable machine learning community.\n\nS2. The main idea of identifying the existence of the failure mode across multiple networks and exploring different choices for characterizing the feature space provides an interesting insight to OOD detection.\n\nWeaknesses:\n\nW1. More discussion of the failure mode for models trained using different baselines and datasets are expected, besides models trained using ERM on DomainBed. \n\nW2. How to quantify the particular failure mode of OoD generalization for discriminative classifiers.\n\nW3. The authors are expected to analysize the relationship between using different backbone architectures, e.g., ViT, and identifying nullspace occupancy.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bgsGceXplI", "reviewer": "Reviewer_mtmw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "===\n\nThis is to acknowledge that I've read the authors' clarifications to me and to the other reviewers. Thank you! I'd like to keep my score as it is; I've done my best to engage with the other reviewers to convince them as to why I think this paper deserves acceptance.\n\n===\n\nTL;DR: The paper hypothesizes a notion of OOD failure, and then presents an empirical technique to isolate it. The paper then demonstrates this failure mode in a real-world setting and presents preliminary ways to mitigate this failure.\n\nMore concretely:\n\n1. The authors hypothesize the notion of \"null space failure mode\" where the presence of non-zero weights assigned to certain directions present in the training domain hurt OOD accuracy.\n\n2. To test this in practice, given a hidden representation, they present a technique for finding a \"basis\" such that the top most directions are the most important for classification under a given distribution. \n\n3. They then show in the DomainBed benchmark that as we project the OOD data only the top few basis vectors (learned using access to), there is surprisingly an _improvement_ in the OOD accuracy. Crucially, this improvement is comparable to SoTA values. This demonstrates the existence of null space failure.\n\n4. To derive a practical algorithm, they propose a similar technique in the leave-one-out domain generalization setting. Here they report that the projection technique shows improvements of ERM (although not surpassing SoTA)\n\n", "summary_of_the_review": "The paper presents an interesting formalization of OOD failure, an area where most work has been empirical and heuristical. The paper supports its claims via a variety of experiments with sufficient baselines/ablations. The core empirical result is interesting/surprising to me. I am hopeful that this will generate valuable follow-up work.", "strength_and_weaknesses": "Strengths\n====\n1. A concrete empirical and formal understanding of _why_ OOD failure occurs is important for developing principled OOD algorithms. The paper presents one such understanding.\n2. The hypothesis of the null space failure mode is simple and intuitive to understand, and to the best of my knowledge not formalized in prior work.\n3. The paper substantiates its claims with a wide variety of experiments on various OOD settings.\n4. It's interesting that a mere projection of the representation (rather than say, reweighting as in [1,2]) results in accuracy improvements.\n4. The paper provides excellent ablations/comparisons making the experiments technically sound. It provides relevant baselines for the projection technique, reports SoTA results for comparison, and also suggests and refutes alternative mechanisms by which the projection technique maybe working  (namely capacity control and l2 regularization).\n5. The projection technique for identifying most important directions maybe of independent interest.\n\nWeaknesses\n===\n\n1. The paper is currently missing key references/discussion connecting it to recent work that has proposed reweighting the top-layer features [ref. 1,2]. \n\n2. My other main complaint is regarding clarity. It's not clear to me how exactly the oracle accuracies are computed: are the bases computed on the target distribution? i.e., is L_train in (3) plugged to be the target distribution loss? I've assumed this to be the case, but I'm also confused if this is simply the original distribution's loss (which wouldn't make sense to me as an oracle). \n- The motivation behind the oracle vs leave-one-out accuracies are presented was confusing and required multiple reads. I'd motivate the oracle accuracy as a theoretical upper bound, and the leave-one-out as a practically achievable value rather than something that \"alleviates a limitation\" of the oracle value.\n- Perhaps a pseudo-code for the computation of oracle accuracy and layer-oracle accuracy may be helpful? \n\n2. It's not clear why the ERM accuracies are different in Table 1 and 2. Perhaps I'm missing something about how ERM is run in leave-one-out settings.\n\n3. I'd have also liked to see what happens to the in-distribution accuracy under these projection techniques. Are there plots for these?\n\n\n[1]: Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization, _Elan Rosenfeld, Pradeep Ravikumar, Andrej Risteski_ https://arxiv.org/abs/2202.06856\n[2]: Last Layer Re-Training is Sufficient for Robustness to Spurious Correlations\n_Polina Kirichenko, Pavel Izmailov, Andrew Gordon Wilson_ https://arxiv.org/abs/2204.02937\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QQNSUhkfHw", "reviewer": "Reviewer_L6zm", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes the observation that some models fail to OoD generalize because discriminative classifiers for OoD test data lie on the null space of learned features. They propose a simple method to avoid this failure: project features onto a low-rank subspace that reflects what was seen in the source data.", "summary_of_the_review": "I think the main observation is interesting and the proposed method makes sense in the context of the observation. I had some conceptual questions about the observation, and would be happy to increase my score if those are addressed.", "strength_and_weaknesses": "Figure 1 is not very intuitively clear to me. I get that because the model was trained on data with v_3=1, its predictions on v_3=10 did not matter, and v_3 is an abstraction of directions of unseen variation in general. However, I’m not sure showing two classifiers side by side is the best way to convey this idea.\n\nExperiments show improvements of about 1~2% OOD accuracy, which outperforms three existing approaches for OoD generalization. The experiment on data poisoning is quite compelling.\n\nQuestion for authors: \n(1) Do you have any hypotheses that would resolve the difference with the gradient starvation phenomenon? Using the notation in page 9, why are features like v_2 ignored while v_3 is attended to (in a suboptimal way, of course)?\n(2) Looking at figs 9-12, many layer/test domain combinations seem to not improve after projection. Do you have a sense for why only some layers benefit from projection, and why the location of those layers is quite different for each test domain?\n\nMinor comments\n- Page 2: your wrapping carried over the the next page, and this page starts with awkward indentation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster