On the Use of Anchoring for Training Vision Models
Authors: Vivek Narayanaswamy
Kowshik Thopalli
Rushil Anirudh
Yamen Mubarka
Wesam A. Sakla
Jayaraman J. Thiagarajan
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Anomaly Detection
OOD Generalization
ML Safety
Anchoring
Deep Neural Networks
Primary area: machine_vision

Abstract: Anchoring is a recent, architecture-agnostic principle for training deep neural networks that has been shown to significantly improve uncertainty estimation, calibration, and extrapolation capabilities. In this paper, we systematically explore anchoring as a general protocol for training vision models, providing fundamental insights into its training and inference processes and their implications for generalization and safety. Despite its promise, we identify a critical problem in anchored training that can lead to an increased risk of learning undesirable shortcuts, thereby limiting its generalization capabilities. To address this, we introduce a new anchored training protocol that employs a simple regularizer to mitigate this issue and significantly enhances generalization. We empirically evaluate our proposed approach across datasets and architectures of varying scales and complexities, demonstrating substantial performance gains in generalization and safety metrics compared to the standard training protocol. The open-source code is available at https://software.llnl.gov/anchoring.

On the Use of Anchoring for Training Vision Models
Vivek Narayanaswamy
Kowshik Thopalli
Lawrence Livermore National Laboratory narayanaswam1@1. 1el1@l11ll..lll gov
Lawrence Livermore National Laboratory thopalli1@11n1. gov
Rushil Anirudh
Yamen Mubarka
Amazon
Lawrence Livermore National Laboratory mubarka1@llnl. gov
rushil55aiirudd@@maal..ccc
Wesam Sakla
Jayaraman Thiagarajan Lawrence Livermore National Laboratory jjthaagrraaanggaan com
Lawrence Livermore National Laboratory sakla1@1in1. gov
Abstract
Anchoring is recent, architecture-agnoste principle for training deep neural networks that has been shown to significantly improve uncertainty estimation, calibration, and extrapolation capabilities. In this paper, systematically explore anchoring as a general protocol for training vision models, providing fundamental insights into its training and inference processes and their implications for general- ization and safety. Despite its promise, we identify a critical problem in anchored training that lead an increased risk of learning undesirable shorteuts, thereby limiting its generalization capabilities. To address this, we introduce a new an- chored training protocol that employs simple regularizer to mitigate this issue and significantly enhances generalization. We empirically evaluate our proposed approach across datasets and architectures of varying scales and complexities, demonstrating substantial performance gains in generalization and safety metrics compared to the standard training protocol. The open-source code is available htttps://oftwwat 11ml. wwl..noo//.//.. noo//nchoring
1 Introduction
Anchoring [ is a recent cebhitctuur-agnossiie principle for training deep neural networks. reparameterizes each input into tuple comprising reference sample and the residual d = i.e., [F, d]. F~ P, and d ~ Pa. Here, P. and PA denote the distributions of references and residuals respectively. The resulting tuple is then fed input deep network instead of the original input x, by concatenating the tuple elements along the feature axis for vector-valued data or the channel axis for image data. Although the first layer of the network needs be modified to accommodate twice the number of input dimensions (due to concatenation). the rest of the model architecture and optimization strategies remain the same as standard training. This simple re- pe-arameterization of the input forces the neural network to model the joint distribution P(r,)) for predicting the target label y. Formally, the training objective can be written as:
E C Cyyyyo Fo ( (FF concat(EE,x ) PoOP (X.))D
0- = arg min
Σ
(I)
D
where L(.) is a loss function such cross-entropy, D is the training dataset and F is the underlying network parameterized by 0. In effect, for a given and reference samples Fı,... Ta. anchoring
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

ensures that Fo (T. ........ ....= F.. (FT del), where de = Fg. In other words, regardless of the choice of reference the model must arrive the same prediction for an input. This principle has been shown to produce models with improved calibration and extrapolation properties 2 3 and to facilitate accurate epistemic uncertainty estimation []. In this paper, we systematically explore the utility of anchoring as a generic protocol for building vision models and make a number of fundamental insights on its training and inferencing, applicability to different architecture families (conv-nets, transformers), and most importantly, the implications model generalization and safety.
Our main contributions in this work can be summarized as follows:
By studying the roles of reference set diversity and the inferencing protocol choice on the behavior of anchored models, we identify critical limitation in current practice. More specifically, we find that conventional anchored training fails to effectively leverage the reference diversity, thus restricting its generalization capabilities, and that merely adopting sophisticated inference protocols [2 cannot circumvent this limitation.
closer look into anchored training and inferencing:
A new anchored training protocol:
We attribute the limited generalization power of anchored models to the increased risk of learning undesirable shortcuts, owing to insufficient sampling of P(.,s) during training, particularly in cases of high reference diversity. To address this, we introduce a new training protocol for anchoring that relies on a novel reference- reference-masking regularizer.
Benchmarking generalization and safety of anchored models: Since anchoring is architecture- agnostic, we benchmark using a variety of nonnnnee/rrnnforrmer architectures on CIFAR-10, CIFAR- 100 and Imagenet- K K datasets. We demonstrate significant improvements in OOD generalization, calibration and anomaly resilience over standard training. We also show that, without incurring any additional training or inference overheads, anchoring is synergistic to existing training strategies (e.g.. data augmentations, optimizers, schedulers).
A Closer Look into Anchored Training and Inference
2.1 What makes anchoring promising training protocol?
Anchored training forces the network to learn a mapping between the joint space of (reference, residuals) and the targets, rather than the original input-target pairs. At first glance, anchoring may seem like trivial reposing of standard training, but it conceptually very different. Through this reparameterization, anchoring creates different relative representations for sample with respect to references drawn from Pr, and attempts to marginalize the effect of the reference when making prediction for that sample. As demonstrated by []]. this process exploits the lack of shift invari- ance in the neural tangent kernel induced by deep networks [4], and implicitly explores a wider hypothesis class that is potentially more generalizable. Furthermore, anchored models have been found to extrapolate better to unseen data regimes through the use of transductive inferencing (2], i.e., identifying optimal reference for each sample, such that the resulting residual is likely to have been exposed to the model during training. While anchoring offers promise, its success hinges on effectively leveraging the diversity of the reference-residual pairs and stably converging for the same protocols from standard training (e.g., architectures, data augmentations, optimizers etc.).
2.2 Does reference diversity play a key role anchored training? ?
A unique property of anchoring is its ability to utilize relative representations w.r.t. a reference distribution P. (realized using reference set R), effectively operating in the joint space P(r,A)- During implementation, the reference set R is defined as a subset of the training data itself i.c.. R C D M. Intuitively, by controlling the construction of R. one can control the diversity of reference-residual combinations that anchored training is exposed to. We hope that with exposure to increasingly large and diverse reference sets, anchoring will explore a wide range of hypotheses, while also ensuring that the model can make consistent predictions for test samples using any randomly drawn reference i E R. However, when the anchored training does not effectively characterize the joint distribution P(t.a): the generalization can suffer, particularly when tested beyond the regimes of training data. To obtain a deeper understanding of anchored training, we conduct an empirical study on CIFAR 10000 datasets by varying the diversity of R.

---

81
Standard Training
Standard Training Anchored Training
Anchored Training
CIFARIOC Avg. Accuracy
Propesed
CITAR1000 o0c Accuracy
Proposed
79
52
51
77
76
50
75
49
74
50
500
2500
5000
50000
5
500 2500
5000
50000
Reference Set Size
Reference Set Size
(b)
Figure 1: Impact of reference set size on anchored training performance. With increase in reference set size, anchoring explores more diverse combinations reference-residual pairs with the hope of demonstrating improved generalization performance. Surprisingly, the existing anchored training protocol does not effectively leverage this diversity even with increased reference set size albeit providing improvements in accuracy over standard training. We propose reference masking, simple regularization strategy for training anchored models that recovers the lost performance.
Setup. We first sub-sample to construct reference sets of varying sizes ranging between 5 and 50000, where the latter corresponds the entire training dataset. The construction is such that each set represents an increasing level of sample diversity (i.c., samples from multiple classes), This is followed by anchored training based the different reference sets with ResNet18 models (5]. All other training specifics and syper-parametedd are fixed across the experiments. Post-training, we evaluate the model performance on the CIFAR10C/100C synthetic corruption benchmarks 6 and report the average corruption accuracy across corruption severity levels.
Observations, Figure la and [b illustrates the performance of CIFARI0/100 anchored training the respective evaluation benchmarks. Interestingly, we observe that the anchoring performance remains fairly similar (minor improvements in accuracy) reference set size. While anchoring provides consistent benefits over standard training (0.5% 1% on average), it is clear that the growing diversity of (t).A)) is not fully leveraged. This observation is in contrary to the insights from existing works, which recommend the use of the entire train data as the reference set for maximal benefits. is also worth noting that we utilize single random reference (from the respective sets) to perform inference. This naturally raises the question if a more sophisticated inference protocol circumvent this limitation that we notice anchored models.
with orders of magnitude growth in the
2.3
Can the choice of inference protocol improve the performance of anchored models?
From existing works on anchoring, we find that different inference protocols can be used to elicit improvements in uncertainty quantification and model extrapolation. For instance, Thiagarajan al. M employed a reference marginalization strategy that samples random references from the reference set to obtain K independent predictions for given input (similar to MC-dropout deep ensembles). This is followed by computing the prediction average along with its standard deviation, wherein the latter different referemce-residual combinations can lead to slightly different predictions for test sample that has not been observed during training, and marginalizing across references offer robustness. On the other hand, Netanyahu et al. [2] introduced the bilínear transduction (BLT) protocol for performing extrapolation from unseen data regimes in regression tasks. It was found that generalizing to an "out of support" (OOS) sample x, (i.e., no evidence of observing such a sample in the training data) can be made more tractable by carefully choosing anchors i ~~ P. such that x; - - d Pa- was argued that, even if the specific combination of [ i] may be observed during training, the anchored model can produce better calibrated predictions when P, and d ~ Pa. This is in contrast to [I], which hypothesized that when the tuple [ x, i] P(r.A)- the inconsistency in the prediction will manifest as epistemic uncertainties. However, neither of these clearly answer the impact of inference protocol choice generalization performance, particularly when the reference
interpreted as estimate of epistemic uncertainty. The intuition is that

---

CIEAR- 1002 Ang. Aucunay
Reloram
Anchaning
Anshoring RM--Ray
1000
Time
randor
50.70
$3.49 -
Mark
K nunãoon
50.70
043
53.55
3,53
om
BLT
50.69 6045
=3.41
24.34
Figure Impact the choice inference protocol on the performance of anchored models [12] (Left) A single random reference is chosen for sample prediction; (Middle) Obtaining predictions using random references followed by averaging: (Right) Bilinear Transduction that identifies the optimal reference for each sample. We find that, while these protocols have varying computational complexities (time (s)/1000 samples), there are no apparent gaps in the performance, indicating that the limitation of anchored training cannot be fixed through sophisticated inference protocols.
set diversity is high. To answer this, we conducted a systematic evaluation of these protocols with anchored models trained on CIFAR 100 with the reference set D.
Setup. We consider three evaluation protocols to make predictions for the CIFARIO0C benchmark (i) I Random, that utilizes a single reference (c.g.. average of samples in R) obtain predictions: (ii) K Random that utilizes K random references followed by reference marginalization (K 10 in our case); (iii) BLT that searches for the optimal reference in R for each test sample. Since conducting such an exhaustive search can be expensive for bigger datasets, we pick subset (set to 50 in our experiment).
Observations, The table in Figure provides the average accuracies obtained from these inference protocols. Interestingly, while these protocols incurs varying inference times (column 3) (BLT >> K random > random), their accuracies are statistically similar to each other (averaged across multiple seeds). This observation implies that that the limitation of anchored training cannot be fixed through sophisticated inference protocols. This motivates us to revisit anchoring and investigate if its behavior can be systematically improved during training itself.
3 Improving Anchored Training via Reference Masking Regularization
close examination of anchored training reveals criti- cal limitation. As the size of the reference set increases, the number of reference-residde ally. For example, when R D. there are ( ((r)) possible pairs, making it impractical to explore all pairs within fixed number of training iterations. This results in insuffi- cient sampling of P(r.a), increasing the risk that anchored training may overlook the reference and make predictions based solely on the residuals. Such non- -eeeraeerll shortcuts are problematic because sample should not be identifiable without considering the reference. There- fore, it is crucial to enhance anchored training by more effectively utilizing the diversity present in large reference sets.
CrossEntropy()
pairs grows combinatori-
for (r, x, in loader:
mask (bernoulli((aphaa)
if nask:
anc CONCATI (8, x-r]
y_hat softmax(model(anc) loss LIU, y_hat)
Unifore Prior on att classes
else:
anc CONCATI x-r))
y_hat softnax(nodel//nn
loss Lly, y_hat)
optinizer.zero_gradl
loss.backward()
optinizer.stepp)
Figure 3: PyTorch style pseudo code for our proposed approach.
3.1 Reference Masking Regularization
We propose a novel, yet simple regularization strategy for improving anchored training. Formally, for a given tuple F r), and a user specified probability that controls how often the training is regularized, reference masking zeroes out the reference and keeps the residual fixed to obtain [0, f). For comparison, the tuple for the same sample x but with "zero" reference (Note: zero vector/image can be a valid reference in our reference distribution) corresponds to [0, x - 0). In order to preserve the integrity the anchoring mechanism, we systematically discourage the model from making meaningful predictions when the reference is masked. This can be implemented by mapping randomly masked tuples to high-entropy predictions (i.c.. uniform probabilities). We achieve this by minimizing the cross-entropy loss between the

---

Standard Training
Anchored Training
Proposed
1.0
1.0
1.0
1.0
Norma Accuracy
0.8
0.6
0.0
0.0
0.0
0.4
0.2
-1.0
0.0
1.0
-1.0
0.0
1.0
-1.0
1.0
Figure 4: Impact of the proposed regularizer on anchored training. Using the CIFARI00C accuracy landscape, i.e., 2D heatmaps the parameter space, we find that our approach identifies flatter and wider optima, thus leading to improved generalization 7
predictions from the masked tuple and the uniform prior U over C' classes (i.e, probability of any class = 1/C). Figure provides the algorithm our proposed approach.
Cireling back to Figure I we observe that the proposed regularization significantly improves general- ization accuracies compared to standard and original anchored training. This clearly demonstrates our strategy's s effectiveness in leveraging the diversity in P.A. Following the insights from the previous section, we use the simple random inferencing protocol to obtain predictions for test samples. At low anchor set sizes ([R) I 50), there is high likelihood of exposing the model to all possible combinations of samples and references, and hence the risk of learning such shortcuts minimal. In such scenario, overemphasizing the masking- based regularization (i.e., high leads to underfitting, as illustrated Figure [I Unsurprisingly, reducing the masking probability can circumvent this underfitting behavior, as evidenced by the original anchored training, where O. However, the benefits our regularization become apparent at larger reference set sizes. Additionally, the table in Figure 2 demonstrates that our approach performs similarly to the original anchored training, thereby implying no discernible impact on the inference efficiency.
Augmentations
Standard Training
Anchored Training
Proposed
CIRAR nor Accorncy
Geometric
33.98
34.43
38.06
RandAug 3
49.74
50.15
53.7
TrivialAug 47.42
47.98
51.22
PixMix 10
58.57
58.38
59.60
Aschored Thinas
(a) Across different augmentation protocols, our proposed regu- larization provides non-trivial gains over standard training. Here, we show the accuracies the challenging case of highest cor-
10%
15%
Label Noisa
(b) Our approach demonstrates improved robustness to label noise in comparison to existing approaches.
ruption severity.
Figure 5: Analysis of Anchored Models. Using evaluations on the CIFAR 100C OOD generalization of ResNet18 models trained on CIFAR100, we study the behavior of the proposed approach when combined with data augmentation protocols (left) and in presence of training label noise (right).
3.2 Analysis
How does the accuracy landscape look like? We hypothesize that the improved generalization of anchoring stems from the training process itself, which inherently enables the model to find better solutions the weight space. To validate this, we follow the analysis in M.] where it shown that that well-genenee gelereeeealllelle solution is typically associated with a wider or flatter local optima in the loss/accuracy landscape. To this end, following the open- open-source implementation from [1], we obtained 2D heatmaps of accuracy evaluated on the CIFAR 100C benchmark over different weight perturbations from the local minima inferring using different training strategies. Figure 4 visualizes the accuracy landscapes, where the and axes represent the co-ordinates that correspond to the different weight realizations. can be observed that our approach produces wider and flatter optima in comparison to the basclines, thus explaining the generalization behavior.
5

---

Can anchoring be combined with data augmentations? Using synthetic data augmentations during training widely adopted method for improving generalization of vision models. In this study, we investigate if anchoring can be utilized alongside existing augmentation protocols, including state-of-the-arr techniques like Mix [0]), and if the observed generalization improvements persist. Table Saaleeaaaaaaa shows the CIFARI 100C accuracies of models trained with different augmentation protocols. Note that, the architecture and the eyper-parameters of the augmentation protocols were fixed to be the same for a fair comparison. Remarkably, our approach consistently provides performance gains regardless of the augmentation protocols used, evidencing its utility as a generic training technique.
Does training label noise impact anchoring? In practice, we construct the reference set R S DD for anchored training. However, under label noise, a fraction (or all) noisy samples can be included in the reference set, and get used for obtaining relative representations. A natural question is if this will impact the anchored training; however, we remind that the tuple construction anchoring does not use the target label of a reference, and the benefits of anchoring will persist even under label noise corruptions. We validate this using the following experiment: We randomly flip the labels of 1% (1 = (0.5, 1, 5, 10, 15, 20}) of training samples before training R model on CIFAR 100, and evaluate the generalization performance on CIFARI00C Figure 5b) illustrates that, with increasing levels of label noise, the anchored models do not demonstrate any additional challenges in handling label noise. In fact, it provides superior generalization (~~ 4% improvements at 20% label noise) when compared to the standard and vanilla anchored training protocols.
4 Experiments
In this section, we empirically demonstrate the effectiveness of our proposed strategy in training models of varying scales (ResNets, Transformers) on datasets of different complexities CI, CIFAR100, ImageNet). We systematically evaluate the generalization of these models under natural covariate shifts and synthetic corruptions. Additionally, we perform a comprehensive evaluation of model calibration, anomaly rejection, and robustness of task adapters in an effort to assess the safety of anchored models. For all experiments in this section, we utilize the entire training dataset as the reference set and train both the original and the proposed anchored models. During inference, we randomly select a single reference from the reference set and perform evaluation on the different test datasets.
Training Datasets. (i) CIFAR-10 and (ii) CIFAR- 100 [3] datasets contain 50, 000 training sam- ples and 10, 000 test samples each of size 32 x 32 belonging to 10 and 100 classes, respectively; (iii) ImageNet-1K 14 is large-scale vision benchmark comprising 1.3 million training images and 50, 000 validation images across 1000 diverse categories.
Architectures. We utilize a suite of vision transformer and CNN architectures with varying levels of structural and parameter complexity. Specifically for training with ImageNet, we consider SWINv2- T 4M params), SWINv2-S (49. 7M), SWINv2-B (87.8M) [15] and ViT-B-16 (86.6M) [16). For CIFAR 100, we use ResNet-18 18 (11.7M) [5] and WideResNNe40-2 M2) 17 architectures, and ResNet- 18 for CIFARIO. We provide the training recipes adopted for our models in Saction/A..
Choice of a. Through extensive empirical studies with multiple architectures, we found using the masking schedule hyper-parameter a 0.2 (corresponds to every 5th batch in an epoch), leads to stable convergence (closely match the top-1 validation accuracy of standard training) on ImageNet and a 0.25 for CIFAR 10/100. Note that, approach performs reference masking for entire batch as determined by a. We have included our analysis on the impact of choice of a in Section A.
4.1 Generalization to Covariate Shifts and Synthetic Corruptions
OOD Datasets and Evaluation Metrics. For models trained on CIFAR10, we evaluate generalization on CIFAR and CIFARIOC. While the former contains 19 different types of corruptions (e.g.. noise, blur, weather, digital), CIFAR10C comprises 10 types of synthetic noise, at 5 different severity levels respectively. Equivalently, for CIFARI00, we use the CIFAR100C and CIFAR 100C benchmarks. For ImageNet-IK, we consider (i) ImageNet-C 6 with 19 natural image corruptions across 5 severity levels, (ii) ImageNet-C [88 with 10 noise corruptions across severity levels; (iii)
ImageNet-R [19] containing different renditions of 200 classes from ImageNet; (iv) ImageNet-S 20

---

Table 1: Generalization performance of CNNs trained CIFAR10/1000 We report the ID test and the OOD (CIFARIO -C/C, CIFARIOO C/C) accuracies of standard and anchored CNNs to evaluate generalization (f). Note, we provide the difference (A) between the proposed and the standard model in each case with blue.
CIFARIVIOI-C Accuracy &
CIFARI0/00-C Accuracy
Dataset
Model
Method
Acc.
Sev.
Sev.
Sev. 2
Sev.
Sev. 5
Standard
95.15
89-44
53.47
77.91
70.74
58.72
86.86
81.97
74.51
65.84
60:31
Vanilla Anchoring
94.92
68.90
84.28
79.16
72.09
59.82
97.04
82.79
75.00
66.73
61.52
CIFAR-10
ResNet-18
Proposed
95.72
90.98
87.15
83.17
77.81
07.26
80.21
85.38
78.31
70.33
65:43
A
10.57
1.54
13:68
15.26
77.07
88.54
12338
3.41
13.83
14.40
5..12
Stundard
77.6
65.56
50.77
51,25
44.57
34.13
02.0
54.08
44.89
36.55
32.27
Vanilla Anchoring
65.67
67.3
52.01
45:27
54.17
36,90
32.72
ResNet-18
Proposed
77.59
67.0
59.51
54.88
48.78
38.06
64.47
58.10
49.78
36.81
CIFAR-100
+0.29
+1.41
+:2.4
+-3.63
+1121
+-1.5
+2.47
-1.02
+-1.89 1.1.89
+-1.87
+1.51
Standard
75.18
62:20
52.82
40.85
40:12
30.05
60.00
52.89
11.44
35.78
31.06
Vanilla Anchoring
76.67
64.55
55.47
49.43
42.84
32.75
61.50
54.42
45.50
36.12
31.11
WRN 40-2
Proposed
77.03
66.0
57.77
52.33
45.61
35.52
03.83
57.70
19.32
00.26
33.29
A
+1.55
+-3.44
4.4.95
45.48
+..32
4.5.4
43.74
-.887
4:4.88
4-4.8
444.33
Table 2: Generalization performance of different transformer architectures trained on ImageNet-IK. We report the ID test and OOD (corruptions and covariate shifts) generalization performance of standard and anchored vision transformers using the topl accuracy. For calibration performance, we report the mean and standard deviation of the Smoothed ECE (4) metric across all ImageNet OOD datasets. Note, we provide the difference (A) between the proposed and the standard model each case with blue.
SWIN22-T (28.4M)
SWIN22-8 (49.7M)
VITbI6 (SGLOM
SWIN22-B (87.8M)
Dataset
Standurd
Proposed
A
Standard
Propused
à
Stundard
Proposed
4
Standard
Proposed
4
ImaggoNet (val)
82.07
62.03
-0.01
83.71
83.68
-0.03
81.07
80.76
-0.31
$1.11
$4.00
-0.02
ImagoNet-R
40.84
41.17
+0.33
45.17
46.03
+1.46
44.06
46.30
+2.33
45.7
48.10 18.16
+2.45
ImageNer-5
27.00 08
27.68
10.00
32.25
30.3
41.00
29.4
33.0
43.66
31.91
33.34
41.43
ImageNet-C (Sev.
71
72:13
+0.50
74.48
74.7
+0.22
72.37
72.52
+0.15
74.45
75.01
+0.70
ImapeNet-C (Sev. 2)
64.5
65.71
+0.52
68.8
60.12
+0332
66.57
67.58
+0.81
68.53 (3.55
69.63
+1.08
ISageNoosCCSev.
57.75
50:21
41.44
02.84
03.65
+0.11
61.6
02.87
41.27
02.31
61.00
41.71
ImageNet::C (Sev. 4)
50:01
+2.14
52.88
+2.25
53.86
56.08
+2.42
ImageNet-C (Sev. 5)
00.00
38.38
+2.52
+1.48
41.09
+3.44
41.87
45.19
+3.32
ImageNos-C (Sev.
73.51
+2.14
+1.20
75.12
ImageNos-C (Sev.
67.
70:46
+3.33
ImageNet-C (Sev.
61.2
65.77
+4.57
+33.30
+-2.39
+3.65
ImageNet-C (Sev,
52.01
57.31
+5.30
58.73
62.05
+4.20
54.7
59.29
+4.50
86
63:53
+4.87
ImageNet-C 5)
46.54
51.76
+5.22
53.7
58.25
+4.05
50.07
54.94
+4486
53.75
58.77
45.02
comprising black and white sketch images from each class of ImageNet. We use the top@1 accuracy to evaluate generalization performance.
Results and Discussions. First, in Table we report the averaged accuracy over all corruptions for every severity level on the CIFAR 10C/C, CIFAR10OCK datasets, for the conv-nets trained on CIFAR 12200/1000 00 respectively. We make key finding that our proposed approach leads to significant gains in corruption accuracies across all severity levels over standard training (1.54% 8.54%) on an average. When compared to CIFARIO, the improvements of anchoring are apparent even at lower severity levels, for e.g., +3.74 improvement with WRN 40-2 at CIFAR severity level 1.
Second, as shown in Table 2 we investigated the efficacy of anchored transformers trained on the turge-scale ImageNet- IK dataset in terms of OOD generalization. It can be observed that our proposed approach consistently yields improvements corruption accuracies over standard training across all architectures. A striking observation is that network capacity plays significant role in effectively leveraging the increased diversity produced by anchored training (we used the entire ImageNet-IK as the reference set). For example, as we move from SWINv2-T (28.4M) to SWINv2-B (88M), we observe increasingly larger performance gains over standard training. Importantly, our proposed strategy handles high noise severity better, achieving improvements of 2% 7% at severity 5 for both
7

---

Imagenet-C and C. All these observations clearly evidence the importance of leveraging the diversity of P(r.A) for enhanced generalization. Finally, we observe from Tables Saalessaana Tand 2 that anchored training
maintains comperitive, and in a few cases, improved ID) accuracies compared to standard training.
4.2 Assessing Safety of Anchored Models
Calibration and Anomaly Rejection. While generalization is key to improve model utility, it must be ensured that the models are not over-confident on unknown inputs and produce well-calibrated prediction probabilities that match the likelihood of correctness. Hence, measuring calibration 221 is vital to understand how tempered the model predictions are under distribution shifts. On the other hand, when the inputs are semantically disconnected and do not share the same label space as the training data, we require the models to appropriately flag them as anomalies. To that end, we also conduct extensive evaluation of model calibration under distribution shifts and anomaly rejection. For the former, we use the ImageNet-C/CC C/CC///S variants, and for the latter, we consider two benchmarks: (a) Vision OOD, comprising commonly used anomaly rejection datasets iSUN (2),, Textures [23 and Places365 [24]; and (b) NINCO 225. recent benchmark containing images with semantic overlap with NetgeNet but with no class overlap. Following standard practice (26), use the Smoothed ECE metric [27] to assess calibration. For anomaly rejection, we obtain the energy scores 26 for both ID validation and OOD data, and report the AUROC metric.
We report the anomaly rejection and calibration performance of of trans- former models trained with ImageNet- IK in Table The results demonstrate notable improvements in anomaly re- jection across architectures, highlight- ing the ability of our approach to bet- ter recognize residuals x, - d ş Pa for an anomalous input sample X and reference observed during training. This is evidenced by sub- stantial gains on both vision OOD and the challenging NINCO anomaly detection benchmarks. For instance, ViTb16 trained with the proposed ap- proach achieves a gain +4.34% on AUROC over non-anchored variant on the NINCO benchmark. In addition, our approach produces consistently lower calibration errors irrespective of the choice of architecture, showcas- ing our ability to produce tempered predictions under OOD shifts.
Table 3: Anomaly rejection and calibration performance of transformers trained on ImageNet-IK. We compare the anomaly rejection performance against standard training us- ing common vision OOD benchmarks (Textures, Places365, and iSUN datasets) and the more recent NINCO dataset. For evaluation, we consider the AUROC (1) metric. Moreover, we also provide Smoothed ECE scores (L) (mean, std) across different Imagenet corruption benchmarks. We highlight the best performing model in each with blue.
Model
Method
Vision OOD
NINCO
Calibration
Standard
76.54
77.46
0.121 - 0.034
SWINv2-T
Proposed
77.65
78.49
0.117 0.027
Standard
77.13
74.73
0.126
0.039
SWINv2-S
Proposed
79.56
78.47
0.119 0.041
Standard
77.20
65.98
0.109
0.037
VTTb16
Proposed
76.88
70.32
0.105
0.028
Standard 75.89
72.13
0.132
0.055
SWINv2-B
Proposed
78.97
74.53
0.124
0.051
Robustness to Task Adaptation. Evaluating model adaptation under task shifts (8 becomes important to shed light onto the quality and re-usability of features inferred in a backbone network. To that end we employ two evaluation protocols: Adaptation (ID Eval) and Adaptation (OOD Eval). In the former, we assume that the distribution of the dataset used for linear probing is the same as that of the test set. In the latter, we first train the linear probe (LP) with our anchored training approach using probing dataset but evaluate the same with data drawn from a shifted w.r.t the probing dataset. Note, for both evaluation protocols, we fix the ViTb16 architecture as the Imagenet pre-trained feature extractor backbone. Note, we set 0.4, a higher value than the original task model training we observed stable convergence.
Adaptation (ID Eval): We consider the following target datasets: (i) CIFAR- 13
(ii) CIFAR- 100 (2 (iii) UCF101 30); (iv) Flowers 102 (31]: (v) StanfordCars [32 The results in Figure a) demonstrate that the proposed approach achieves substantial performance gains over the baseline (0.81% 2.68%). These findings suggest that the reference masking regularizer yields feature representations that are transferable even under complex task shifts.
8

---

VTTb (86.6M)
Evaluation
Trin Domain: Real
Train Domain: Sketch
Dataset
Standard
Proposed
Domain
Standard
Proposed
Standard
Proposed
CIFAR-I0
95. 18
96.29
+0.81
Real
41.35
44.81
13.46
CIFAR-100
80.1
82.78
+2.68
Sketch
25.85
25.02
12:17
UCF101
75.55
77.01
+1.46
Clipart
37.38
38.98
11.6
35.4
37.76
12.36
Flowers102
91.68
95.7
+1.02
Painting
46.3
46.97
10.87
31.42
32.7
1.228
StandfordCars
58.54
61.15
+2.61
(b) OOD Evaluation of LP Adaptation. Using the ViTbl6 backbone we train two LPs the Real and Sketch domains from the Domainnet dataset respectively. We then assess their zero-shot accuracies on three held-out test domains. Our findings show that the proposed approach consistently outperforms the non non-anchored baselines.
(a) LP-based adaptation for ViTb16 ar- chitecture pre-trained on Imagenet- 1K on downstream tasks. We measure the accu- racy (1) the adapted model using the vali- dation split the target dataset.
Figure 6: Assessing anchored and standard pre-trained ImageNet backbones robustness to task shifts.
Adaptation (OOD Eval): For training linear probes, we use the DomainNet [33,, comprising of images from 345 categories across six diverse domains. Specifically, real, sketch, clipart, and painting and train probes on (i) images from the real domain, and (ii) images from the sketch domain respectively. We then evaluate the LPs on the remaining three held-out domains. As Figure b(b) illustrates, our proposed reference masking continues to substantially outperform standard training baseline on all held-out domains under both configurations. We attribute this behavior to our approach being able to effectively leverage the diversity in the reference-residual space produce robust and better generalizable features supporting transferability.
pick four domains, namely
5 Related Work
Anchoring in Predictive Models. Our work is based on the principle of anchoring first introduced in [ where was used achieve stochastic data centering for epistemic uncertainty estimation. Since then, the anchoring has been extended to variety of use-cases and applications. For e.g, Netanyahu et al. (2) utilized anchoring for extrapolating to unseen data regimes [2] in regression settings and Trivedi et al. (44] employed the same for graph neural network calibration. In contrast, our paper the first to explore and facilitate the utility of anchoring as a viable training protocol for large scale vision models.
Data Augmentations. Augmentation strategies enforce models to be robust under different pixel- space manipulations improving generalization. For e.g., strategies such as Augmix (35) or random convolutions (RandConv) 36 are known to improve generalization. Recent advancements in the field include strategies such as PixMix [10), which utilizes an external dataset with complex image patterns augment the training data, and ALT [37], which learns adversarially robust augmentations. While the idea of enforcing prediction consistency in anchoring might appear similar to training with synthetic data augmentations, we emphasize that anchoring does not alter the data (e.g., with perturbations or geometric transformations) but only creates relative representations for each sample with respect to different reference choices. Furthermore, can be combined with data augmentations to achieve further gains in generalization (Table 5a).
Model Safety. As models are being increasingly adopted in variety of sensitive applications safe model deployment has become critical 440 41. In this context, generalization to data beyond
the training distribution 442 6. ability to accurately detect anomalies in the input data [4) 26] 44 as well producing calibrated prediction probabilities [21 are all important facets of safety evaluation. Hendrycks el al. [00 argued that most existing training strategies compromise for one safety objective to satisfy another objective, thus limiting their real-world utility. We find from our experiments that anchoring jointly produces better generalization, calibration and anomaly rejection properties, which makes it promising choice for practical deployment.

---

Conclusion
Through this work, we showed that anchoring leads to significant performance gains generalization and other safety metries, including calibration, anomaly rejection, and task adaptation, across varying dataset sizes (CIFAR-I0 to ImageNet) and model architectures (Conv-Nets to Transformers). Notably, when the training recipe includes high-capacity architectures or advanced mechanisms, our method yields even greater performance gains over the base models. Our observations suggest that anchored training with larger reference requires reference masking regularization control the risk of learning undesirable shortcuts while making predictions. However, we realize that state-of sate-off-hhear results in OOD generalization are often obtained using model souping (45) by fine-tuning large scale pre-trained models 46]. Hence, we believe it will be valuable to integrate anchoring into these approaches. While we have not theoretically characterized the generalization of anchored models, our hypothesis is rooted in existing theory and our empirical results provide evidence for the hypothesis. Finally, must be noted that anchoring is domain domain-agnosticc, training strategy for deep neural networks. However, developing theoretical understanding of anchored models and understanding its benefits domain-specific applications is crucial and forms an important future direction.
sechitecture-agnostee and task agnostic
Acknowledgements
This work was performed under the auspices of the U.S. Department of Energy by the Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27334 Lawrence Livermore National Security, LLC. and was supported by the LLNL-LDRD Program under Project No. 22-ERD- 006. LLNL-CONT-864979)
References
[1] Jayaraman J. Thiagarajan, Rushil Anirudh, Vivek Narayanaswamy, and Peer timo Bremer. Single model uncertainty estimation via stochastic data centering. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL httpppp////pppeeeeeeeeeeee/e///a
[2] Aviv Netanyahu, Abhishek Gupta, Max Simchowitz, Kaiqing Zhang, and Pulkit Agrawal. Learn- ing to extrapolate: A transductive approach. In The Eleventh International Conference on Learn- ing Representations, 2023. URL https::/oppnnrevee...tt wwt.:eewwwwwweeeeeeeeeeeeeeeeeee wttt:///ooooooomediiiieae
[3] Rushil Anirudh and Jayaraman Thiagarajan. Out of distribution detection via neural network anchoring. In Asian Conference on Machine Learning, pages 32-47. PMLR, 2023.
[4] Arthur Jacot, Franck Gabriel, and Clément Hongler. Neural tangent kernel: Convergence and generalization in neural networks. Advances in neural information processing systems, 31, 2018.
[5] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[6] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019. URL httl:tttss://oeereeviii wttpriiwwwneeeeeeeeeeeeeeeeeeeee wttp://oooooondddddrtt:1::-1111-
[7] Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry P Vetrov, and Andrew G Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. Advances neural information processing systems, 31, 2018.
[8] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition workshops, pages 702-703, 2020.
10

---

[9] Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the data augmentation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 774-782, 2021.
[10] Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt, Pixmix: Dreamlike pictures comprehensively improve safety measures. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16783-16792, 2022.
[11] Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv: 05407, 2018.
[12] Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Neural Information Processing Systems, 2018.
[13] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[14] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 1 115(3)2211225 2015.
[15] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning. Yue Cao, Zheng Zhang, Li Dong. Furu Wei, and Baining Guo. Swin transformer v2: Scaling up capacity and resolution. In International Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021. URL http:pppwweeeeeeeeeeeeeerroaaaaa
[17] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference 2016. British Machine Vision Association, 2016.
[18] Eric Mintun, Alexander Kirillov, and Saining Xie. On interaction between augmentations and corruptions in natural corruption robustness. Advances in Neural Information Processing Systems, 34:3571-3555 2021.
[19] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustmess: A critical analysis of out-of-distribution generalization. ICCV, 2021.
[20] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric Xing. Learning robust global represen- tations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages (0506-10518. 2019.
[21] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pages 1321-1330. PMLR, 2017.
[22] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 3485-344400 3492. IEEE, 2010.
[23] Mircea Cimpoi, Subhransu Maji, Jasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[24] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEFE transactions on pattern analysis and machine intelligence, 4006):1452-1466 2017.

---

[25] Julian Bitterwolf, Maximilian Mueller, and Matthias Hein. In or out? fixing imagenet out-of- distribution detection evaluation. In ICML, 2023. URL http::/t////ro .mw.inns...m.....mmmm.......e..m press/ #202/bitterwo1f f2aa. html
[26] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. Advances in Neural Information Processing Systems, 2020.
[27] Jaroslaw Blasiok and Preetum Nakkiran. Smooth ece: Principled reliability diagrams via kernel smoothing. arXiv preprint (xXX:::3099.2236. 2023.
[28] Anders Andreassen, Yasaman Bahri, Behnam Neyshabur, and Rebecca Roelofs, The evolution of aut-of-distribution robustness throughout fine-tuning. arXiv preprint wrxiv:210.11581/ 2021.
[29] Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http:///ww. cs. toronto. edu/kriz/cifar html, 55:5, 2014.
[30] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 2012.
[31] Maria-Flena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008.
[32] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop on 3D Representation and Recognition (3dRR-13) Sydney, Australia, 2013.
[33] Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 1406-1415, 2019.
[34] Puja Trivedi, Mark Heimann, Rushil Anirudh, Danai Koutra, and Jayaraman J. Thiagarajan. Estimating epistemic uncertainty of graph neural networks. In Data Centric Machine Learning Workshop @ ICML, 2023.
[35] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[36] Zhenlin Xu, Deyi Liu, Junlin Yang. Colin Raffel, and Mare Niethammer. Robust and generaliz- able visual representation learning via random convolutions. In International Conference on Learning Representations, 2021.
[37] Tejas Gokhale, Rushil Anirudh, Jayaraman Thiagaraaan, Bhavya Kailkhura, Chitta Baral, and Yezhou Yang. Improving diversity with adversarially learned transformations for domain generalization. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 434-443, 2023.
[38] Thomas Davenport and Ravi Kalakota. The potential for artificial intelligence in healthcare. Future healthcare journal, 6(2):94, 2019.
[39] Daniel Bogdoll, Maximilian Nitsche, and J Marius Zöllner. Anomaly detection in autonomous driving: A survey. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4488-4499 2022.
[40] Clark Barrett, Brad Boyd, Elie Bursztein, Nicholas Carlini, Brad Chen, Jihye Choi, Amrita Roy Chowdhury, Mihai Christodoreseu, Anupam Datta, Soheil Feizi, et al. Identifying and mitigating the security risks of generative ai. Foundations and Trends in Privacy and Security, 6(1):1-52, 2023.
[41] Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint arXiv:2109.13916 2021.
12

---

[42] Jingkang Yang, Kaiyang Zhou, Yixuan Li, and Ziwei Liu. Generalized out-of-diitributtin detection: A survey. arXiv preprint arXiv:2110.11334 2021.
[43] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. Proceedings of International Conference on Learning Represen- tations, 2017.
[44] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference Learning Representations, 2018.
[45] Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo- Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights multiple fine-tuned models improves accuracy without increasing inference time. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari. Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23965-23999 PMLR, 17-23 Jul 2022. URL https: httpts://ppoceeeeeeeeeeee http:///oooeeeeneeeeeeeee mal.prrss. v162/wortsman222 11ttttmmmn22a.hmll html
[46] Sachin Goyal, Ananya Kumar, Sankalp Garg, Zico Kolter, and Aditi Raghunathan. Finetune like you pretrain: Improved finetuning zero-shot vision models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 2023.
June
[47] Rushil Anirudh and Jayaraman Thiagarajan. Out of distribution detection via neural network anchoring. In Asian Conference on Machine Learning (ACML). PMLR, 2022.
13

---

A Appendix
A.1 How does the choice of impact training?
The parameter controls the frequency of the regularization applied to anchored training. Under the assumptions of operating with wide reference sets, through Table we note that moderate to small values of a enable better regularization of anchored training. Notably, setting a = 0.25 i.c. masking references for one in four samples, yields impressive gains ID and OOD performance. Conversely, over-regularizing by setting a to large value (e.g., 1.0) entails masking every reference, unsurprisingly results in models that generalize poorly, as they are tasked with learning solely from residuals.
Table 4: Impact of a on anchored training. As we gradually increase a, there is a risk of over- regularization which can lead to severe underfits. Note, we consider R D in this study.
0.0
0.25
0.5
0.75
1.0
ID Test Ace. %
77.21
77.89
76.97
75.4
57.90
OOD Acc. %
51.01
53.77
52.61
52.30
35.40
A.1.1 Choosing a in practice
At low reference set sizes, there is a high likelihood of exposing the model to all possible combina- tions of samples and references, and hence the risk of learning shortcuts is minimal. In this case, overemphasizing the reference masking probability (i.e., increasing a) can significantly inhibit this exposure. Consequently, this leads to underfitting as the model is tasked with learning solely from the residuals which is undesirable in practice (Blue curve for reference set size <50 in Fig. D Reducing can combat this behavior, as evidenced by the original anchored training (special case of reference masking with
0 namely the red curves for reference set size $50 in Fig, D
Now, with larger reference sets (e.g., datasets in the scale of ImageNetIK), the number of reference- residual pairs grows combinatorially, making it impractical to expose the model to all diverse pairs in fixed number of training iterations. In such a scenario, reducing a can increase the risk of learning shortcuts and lead to suboptimal performance. Increasing a on the other hand can in fact aid training as it systematically avoids these shortcuts and improves generalization. In summary, the optimal value depends both on the reference set size and the convergence behavior of model training.
A.2 Does Training for Additional Epochs Alleviate the Reference Set Size Problem?
One possible way of alleviating this problem is by redueing the reference set size. However, this reduces the diversity of the reference-residual pairs exposed during training and can lead to a poor solution. While the issue of diversity can be combated with large reference set sizes, increasing the number of epochs alone does not solve the prohlem as there exists a combinatorially large number of reference-residual pairs which cannot be practically explored, and the model will still be vulnerable to shortcuts. Moreover, modifying the number of training epochs results in non-trivial modifications in the training dyper-parameter (e.g., learning rate schedules) and can lead poorly convergent models if the hyper-parameters are chosen incorrectly. Hence, our reference masking regularizer for anchored training, helps mitigate shorteut decision rules while also being computationally efficient.
A.3 Additional Details on Training Protocols
the recipes (augmentations, epochs, optimizers) leveraged for model training. Note that, the other hyper-parameters can be found in [47] for CIFAR10/100 and httpss//pytoorc htmmllll..mm........mmm.mmm for ImageNet. We emphasize that, anchoring can be used as generic model training wrapper, allows integration with any data augmentation or training strategy, and is not restricted to the recipes considered.
14

---

Table 5: Protocols adopted for training anchored models across different datasets and architec- tures. While we adopt standard training recipes for training our models, we note that anchoring can serve generic wrapper that can be applied on top of any other existing recipe.
Numbhr Nen-Anchaned Anchreed
Model
Datert
Training Rocipes
Optimter
RasNes WRN-40-2
IVV/0
50D Mali
S.
Insgchitt
Steap, Dutitiue, Astoligeeci, Bandon Erase Augftin, Lubsl Saweking
Adaar Arscaling
IrogoNes
Wivap, Cuelite, Resodegears, leghtss, Label Sneothing
300
390
AraetW wrth Codne Areesting
15

---

A.4 Expanded ImageNet Generalization Results
We provide expanded version of Table 2 that includes the anchored training protocol without the reference-masking regularizer.
Table 6: Generalization performance of models trained ImageNet-IK. We compare the generalization performance of different training strategies under both ID and OOD (corruptions and distribution shifts) settings. For evaluating the prediction performance on each of the benchmarks, we consider the widely adopted Top1 accuracy metric. For calibration performance, we report the mean and standard deviation of the Smoothed ECE (4) metric across all ImageNet OOD datasets. Note, we highlight the best performing model in each case with blue.
ImaggeNee-C
IrangeNe.CC
ImageNet-RR
IusgeNees
Stanked
82.00
90.54
27.05
11.45
66.80
$0.97
47:77
35.46
11.37
67.12
01.2
$2.01
45.54
0.121
SWIN22-
Propoce
82.08
41.17
27.68
72.15
66.71
59:21
50.01
38.55
73.51
70.45
05:77
57 21
51.76
0.117 0.027
Sundard
83.71
65.17
$1.20
67.14
0.126 0.000
SWINGOS
Archoing
$4.0
45.55
22305
74.75
08.87
08.12
64.7
41.34
76.07
73.35
08.79
50.43
03.19
0.122 0.06G
Sundard
$1.07
61.00
70.1
71.37
09.57
01.66
$2.58
11.00
77.75
02100
031
54.7
50:07
0.100 0.00
(56655
Anchening
80.57
43.35
50.52
72.64
67.14
69.03
$1.46
13.21
09:21
65.57
56.00
51.46
0.006 0.035
Proposed
89.76
06.39
33.0
71.33
07:38
02:88
53.
73.065
70.99
06
59
$4.04
0.105 0.024
Staraked
$4.11
66.7
81.11
14.45
68.65
62.84
53,50
41.87
15.12
12.15
67.15
$5.60
63.75
0.152 0,056
03.42
00.25
52.08
0.129
Propoced
84.00
48.16
53.34
76.24
09.83
04.05
$6.05
46.19
77.1
74.60
70.81
63. 08
68.17
0.124 0.005
A.5
Expanded Anomaly Rejection Results for Vision OOD Datasets
While Table B in the main paper provided anomaly rejection results averaged over all Vision OOD datasets, we expand and present metries for each dataset in Table 7
Table 7: Measuring anomaly rejection performance on Imagenet-IK. We report the AUROC (1) scores
Anomaly Rejection (AUROC)
Architecture
Method
iSUN
Textures
Places365
Standard
80.25
76.83
72.53
SWINv2-T
Anchored Training
78.68
76.64
74.75
Proposed
77.69
78.09
77.16
Standard
82.89
77.87
70.63
SWINv2-S
Anchored Training
87.73
80.83
76.67
Proposed
84.18
79.66
74.85
Standard
86.92
79.24
65.72
VTTb16
Anchored Training
85.17
76.88
66.16
Proposed
84.55
78.91
67.18
Standard
85.32
76.35
65.99
SWINv2-B
Anchored Training
85.98
77.88
70.75
Proposed
87.34
75.74
73.66
16

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In a few cases of training transformers on ImageNet, our approach produces slightly lower n--distrbbttinn accuracies (for e.g, 0.3% reduction VitB 16). While this comes at a significant gain in OOD accuracy, the question remains of how to improve the anchored training to prevent this performance drop. We also provide additional limitations section in conclusion section.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: [NA]
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have provided extensive details regarding data pre-proceee ve-processing pipelines, hyper-parameters, training protocols in 4 We have also provided Py PyToh code snippets for easy implementation of our approach.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: [NA]
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
We used standard train, validation and test splits that are made available with datasets. Training protocols, hyperparameters and the sensitivity of model for hyperparame- ters are provided the appendix.
Justification:
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Experiments were conducted using multiple seeds and error bars reported.
8. Experiments Compute Resources
17

---

Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: [NA].
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http://ppp/....pp...p..o/ooottta
Answer: [Yes]
Justification: [NA]
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed them in the related work as well conclusion section. In opinion, there are no negative societal impacts.
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer:[NA]
Justification: [NA]
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: [NA]
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
18


Reviews:
[{"review_id": "7wqzz6f0K0", "reviewer": "Reviewer_VjYP", "summary": "The authors analyze the effect of anchored training through a series of small experiments and find that, contrary to claims in prior works, increasing the size of the reference set is not beneficial and that this shortcoming cannot be mitigated through existing inference strategies. The authors provide a simple yet efficient fix by randomly masking out the reference during training, and forcing the model to make high entropy predictions in those cases. This solution does not incur any training overhead, and the authors demonstrate in extensive experiments that the fix is applicable to different models and datasets, yields improvements for OOD performance over various distribution shifts, and improves calibration and anomaly resilience.", "questions": "1. Tab. 2 Do you have any insights why the improvements on ImageNet-S and ImageNet-R are drastically different for SWIN transformers and ViT?\n2. (minor) The formatting of paragraph headers in the introduction is weird and inconsistently using underline.\n3. (minor) Erroneous comma in L.165\n4. (minor) It is hard to visually assess from Fig. 4 whether the optima are significantly different.", "limitations": "Limitations were sufficiently addressed, especially the empirical nature of the work.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is very well written and structured and is overall easy to follow. The initial experiments highlight the studied problem well.\n2. The authors showcase an important limitation to existing anchoring techniques that was unknown to the community.\n3. The proposed solution is simple and is demonstrated to consistently improve performance across models and datasets.\n4. The experiment section is extensive and covers both OOD performance as well as safety-relevant metrics. The results convincingly demonstrate the effectiveness of the proposed method.", "weaknesses": "The paper is very well written, I don't see any major weaknesses that would prevent an accept.\n\nMinor weakness: The optimal $\\alpha$ is determined when using the entire dataset as a reference set. However, as is clear from the motivation, risk of spurious shortcuts is larger with a smaller reference set. Wouldn't this imply that the optimal $\\alpha$ would be larger for smaller reference sets? How should this value be chosen in practice and for datasets larger than ImageNet-1k?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive feedback. Here are our responses to your questions. We plan to incorporate some of these clarifying comments to the manuscript as well.\n\n**1. Choice of $\\alpha$** \n\nWe want to clarify that at low reference set sizes, there is a high likelihood of exposing the model to all possible combinations of samples and references, and hence the risk of learning shortcuts is minimal. In this case, overemphasizing the reference masking probability (i.e., increasing $\\alpha$) can significantly inhibit this exposure. Consequently, this leads to underfitting as the model is tasked with learning solely from the residuals which is undesirable in practice (Blue curve for reference set size $\\leq$ 50 in Fig.1 of the paper). Reducing  $\\alpha$ can combat this behavior, as evidenced by the original anchored training (special case of reference masking with $\\alpha = 0$, red curves for reference set size $\\leq$ 50 in Fig.1). \n\nNow, with larger reference sets (e.g., datasets in the scale of ImageNet 1K), the number of reference-residual pairs grows combinatorially, making it impractical to expose the model to all diverse pairs in a fixed number of training iterations. In such a scenario, reducing $\\alpha$ can increase the risk of learning shortcuts and lead to suboptimal performance. Increasing $\\alpha$ on the other hand can in fact aid training as it systematically avoids these shortcuts and improves generalization. In summary, the optimal $\\alpha$ value depends both on the reference set size and the convergence behavior of model training.\n\n**2. Performance Improvements on ImageNet-R/S with VITb and SWINv2B**\n\nAnchored training with ImageNet-1K involves exposing the model to significantly large and diverse reference-residual combinations. Our results show that, in order to model such a large joint distribution and to leverage the diversity of the reference set, along with the proposed masking regularizer, higher capacity networks (VITb & SWINv2B) are also required. We hypothesize that this behavior helps such networks better handle challenging, far out-of-distribution datasets such as ImageNet-R/S. For instance, we observe an average of 1.5% and 1.7% improvements in ImageNet-R and S accuracies respectively when using such architectures over lower capacity models. \n\n**3. Formatting**\n\nWe will make changes to the paragraph header, correct underline inconsistencies in the introduction and improve Fig. 4 in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1lSyh74jXd", "reviewer": "Reviewer_fpNo", "summary": "This paper presents a thorough discussion on the use of anchoring for training vision models. In particular, the paper tackles 1) the problem of reference diversity when training with anchoring to explain how superior generalization can be achieved 2) addresses the problem of spurious correlations learnt between the residual and 3) how different inference-time strategies can enable greater out-of-support generalization. Overall, this comprehensive study of anchoring provides useful guidelines for how anchoring should be applied to extract maximum performance. The paper empirically confirms this via the proposed anchoring scheme outperforming prior work noticeably.", "questions": "Have the authors compared the out of support generalization of anchoring procedures to other methods for domain generalization (which tackles a similar problem)? Considering datasets and baselines from *In Search of Lost Domain Generalization* (https://arxiv.org/abs/2007.01434) can further broaden the impact of this paper.", "limitations": "N/A", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1) Clarity: The paper is very clearly written and easy to follow. Readers unfamiliar with the literature like myself are able to understand what anchoring is, how it can be useful for (out of support) generalization and how current methods fail to apply anchoring in the most effective way. \n\n2) Thoroughness of Evaluation: The paper conducts thorough ablations on several components of the anchoring pipeline. Reference diversity, reference masking, inference procedure etc. More", "weaknesses": "No obvious weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive comments and feedback. We hope our response addresses your concern.\n\n**Domain Generalization Benchmarks**\n\nThank you for this question. We would like to highlight that we performed experiments on DomainNet which is one of the benchmarks from DomainBed [1] (Line 293 of the main paper).  Following the setup in [1], we trained on one of the domains (source) and evaluated the model on the remaining domains (target).  While it is common in the domain generalization literature to train models end to end on the source dataset, we instead trained a linear probe on top of an anchored feature extractor pre-trained on ImageNet. This was motivated by the need to investigate the impact of anchored training in producing better generalizable feature extractor backbones.  In particular, we trained a linear probe with ERM [1] using the ‘real’ and ‘sketch’ (source) splits respectively from DomainNet. We then evaluated performance on the other (target) domains and observed performance improvements over the non-anchored variant.  \n\nWith respect to the domain generalization specific baselines and training strategies (e.g., DRO, IRM) used in [1], we would like to emphasize that our proposed anchored training protocol can be simply used as a wrapper on all such methods and we expect it to improve overall performance similar in spirit to our analysis on augmentation methods in Section 3.2 (Fig 5a of the main paper). We plan to perform an extensive analysis on domain generalization as a part of our future work. \n\n\n[1] Gulrajani, Ishaan, and David Lopez-Paz. \"In Search of Lost Domain Generalization.\" ICLR 2021", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GXj3uKGrWW", "reviewer": "Reviewer_Tq8Q", "summary": "In this paper, the authors propose a new strategy to train anchoring-based models, significantly improving performance, training efficiency, and model generalization compared to previous approaches. The key to the method is the added masking strategy that allows the model to better profit from anchoring-based training. The authors demonstrate that modifications only in inference (using several samples or searching for the best references) or the number of used references do not improve model performance, while the application of the masking procedure significantly improves it, as shown on various image classification datasets, specifically CIFAR-10, CIFAR-100, and ImageNet, using different architectures (both CNN and attention-based). The experiments demonstrate the effectiveness of the proposed method and the significant benefit of using it for improved generalization.", "questions": "* Why do the authors focus on vision models when the method seems to be very generic and applicable to other domains as well?\n\n* One of the claims the authors make is that the proposed masking procedure helps with the problem of the model ignoring reference input. They support this claim with, for example, Figure 2—an experiment showing that without this masking, we do not observe improvements in terms of performance, which is only a proxy for the claim. Is it possible to measure the sensitivity of the model with regard to reference inputs (for example, by adding noise to it and measuring the change in the outputs)?\n\n* As far as I understand from the method description, the final method in Section 4 uses only one reference image for inference. How does the performance change with an increased number of references? The lack of improvements in performance (e.g., as in the right plot in Figure 2) seems strange to me since we would observe the opposite behavior in all existing ensembling approaches (e.g., [1, 2, 3, 4]). How would one explain such behavior? Additionally, it would be good to see some comparisons with these methods or at least include them in the discussion. \n\n[1] Lakshminarayanan, Balaji, Alexander Pritzel, and Charles Blundell. \"Simple and scalable predictive uncertainty estimation using deep ensembles.\" NeurIPS 2017\n\n[2] Wen, Yeming, Dustin Tran, and Jimmy Ba. \"Batchensemble: an alternative approach to efficient ensemble and lifelong learning.\" ICLR 2020\n\n[3] Durasov, Nikita, et al. \"Masksembles for uncertainty estimation.\" CVPR 2021\n\n[4] Laurent, Olivier, et al. \"Packed-ensembles for efficient uncertainty estimation.\" ICLR 2023", "limitations": "No", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The paper is clearly written and easy to follow. The idea is intuitive and easy to grasp. The related work section provides an adequate discussion of existing approaches to anchoring-based training. The analysis narrative, with the presented drawbacks of existing methods, is very clear and easy to understand.\n\n* The idea of masking the reference input argument is very clear and logical. The intuition behind why the problem could occur: 1) argument size grows combinatorially, and therefore 2) the model could learn to ignore the reference argument; seems correct, which is further clearly supported by the experiments.\n\n* The authors provided an extensive evaluation of their approach, spanning different datasets and architectures, which provides a solid grounding to support the proposed method.", "weaknesses": "* It seems that the evaluation could benefit from an additional comparison with other existing state-of-the-art OOD/uncertainty methods to better represent the quality of the results (not just in comparison with former anchoring-based approaches, but overall).\n\n* From the perspective of the experimental evaluation, I would be curious to see evidence that the behavior demonstrated in the paper would hold in other domains, such as texts, graphs, more complicated vision tasks (e.g. segmentation), not limiting to image classification task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive feedback. We hope our responses address your questions.\n\n**1. Generic Applicability of Anchoring**\n\nThank you for this question.  We concur with you that anchoring is a protocol for training deep neural networks for use with any domain for any application (e.g., texts, graphs or vision tasks such as segmentation). In this paper, our goal was to identify the shortcomings of the original anchored training [1] and develop algorithms to improve the same, and vision was chosen only as a domain of convenience. Though we have performed initial experiments on using anchored training for other data domains (e.g., graphs, text), we did not include them to restrict the scope of the current submission. We find that our proposed anchoring approach produces performance improvements even in non-vision tasks. In the final version of the paper, we will include this discussion as part of the concluding remarks. In summary, anchoring is a domain-agnostic, architecture-agnostic, and task-agnostic training strategy.\n\n**2. Clarifying ‘Model Ignoring Reference Input’**\n\nIt must be noted that the anchoring principle as demonstrated in [1] implicitly explores a large family of functions during training due to the lack of shift invariance of the underlying neural tangent kernel when the input is translated by a reference. We believe that this strategy produces a local optima similar in spirit to _stochastic weight averaging_ [2] that averages multiple solutions along the trajectory of gradient descent. Unique to anchoring, the quality of the converged optima depends upon the diversity of the reference-residual pairs (induce a rich family of functions) exposed during training. \n\nHowever, we observe that the original anchored training even with large reference sets does not fully leverage the reference-residual diversity and converges to a poor local optima (Fig. 4b). We attribute this to the anchored model relying on shortcuts to make predictions. Please note that, the usage of the term shortcut (ignoring the reference) in the context of anchoring is different from convention.  Shortcuts manifest during anchored training when the model can predict well only with certain arbitrary references but on an average converges to a poor optima.  Basically, the functions induced by ignoring the reference are entirely different from the ones obtained without ignoring them, making the model eventually converge to a sub-optimal (implicitly averaged) local optima.  We will better clarify this in the final version of the paper. \n\n**3. Clarifying the Anchoring Inference Mechanism**\n\nWe would like to emphasize that anchored training (i) enforces prediction consistency of a sample with any reference; (ii) produces a single model; and (iii) converges to a local optima in a manner akin to stochastic weight averaging [2]. Moreover, when the diversity of the reference-residual pairs is well leveraged during training, it allows the model to converge to a wider optima improving model generalization (Fig. 4 in the main paper). Therefore, the `quality’ of the optima governs performance during inference and as a result the inference strategy (e.g., choosing K random references for inference) does not alter the (mean) model performance. It must be noted that anchoring must not be viewed under the lens of model ensembles that train multiple models where each member explores different yet possibly diverse local optima.  While [1] measures discrepancies in predictions of a sample with different references as a notion of epistemic uncertainty, we find that the mean performance does not change. Fig. 2 in the main manuscript compares different inferencing protocols (1 random anchor, K anchors and transduction) and finds no significant differences in accuracy.\n\n**4. Comparison with Existing OOD/Uncertainty Estimation Methods**\n\nThank you for this very important question. While a systematic evaluation of OOD detection with state-of-the art methods is most essential, it is beyond the scope of this paper. Our paper aims to establish anchoring as a useful training protocol and demonstrate its efficacy across a spectrum of tasks and model architectures. Although [1] provided evidence of the efficacy of epistemic uncertainties from anchoring for OOD detection, we will be conducting a large scale study as a part of our immediate future work.  \n\n\n[1]  Thiagarajan et al.  Single model uncertainty estimation via stochastic data centering, Neurips 2022\n\n[2]  Izmailov et al. \"Averaging weights leads to wider optima and better generalization”, UAI 2018", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T9jOZdfGXJ", "reviewer": "Reviewer_8YxY", "summary": "This paper identifies a major problem with anchored training, that the performance of anchored training does not increase with increasing reference set size, and proposes a simple regularization approach to overcome this problem. This approach is evaluated on OOD generalization, calibration and anomaly rejection, and task adaptation, and various facets of anchored training are analyzed.", "questions": "1. Is there any explanation for why the accuracy stays relatively constant (Figure 1) regardless of reference set size?\n2. Does training for more epochs help alleviate the reference set size problem?", "limitations": "The limitations of this work are discussed by the authors at the end of the paper. Negative societal impact is probably not a concern for this paper.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes the interesting finding that the performance of anchored training does not increase with increasing reference set size, and that this problem is not alleviated by more sophisticated inference strategies. The paper also proposes a simple reference-masking regularization technique to help alleviate this problem. The experiments show the effectiveness of the proposed approach, and there is also analysis of how the method interacts with data augmentation and noisy labels. An ablation study of the $\\alpha$ parameter is also performed. Training recipes are also provided, making the paper easier to reproduce.", "weaknesses": "One weakness is that the reference set selection strategy and reference set sizes are not explained for the experiments. \n\nThe impact/novelty is a bit limited because of the lack of comparisons to non-anchored training works.\n\nMinor points: in the tables, decreases in performance could be colored in a color other than pink. Figure 1 could be improved with error bars. One highlighting was missed in Table 3. The abbreviation LP is not defined.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank you for your positive feedback. We hope our responses address the questions you have raised.\n\n**1. Accuracy Remains Constant Regardless of Reference Set Size**\n\nWe would like to clarify that this is precisely the problem with the original anchored training protocol, which we solve in this paper. As shown in Fig. 1 of the paper, the original anchoring protocol does not fully leverage the diversity of reference-residual pairs with increasing reference set size and maintains a relatively constant accuracy regardless of the reference set size.  We hypothesize that this is because as the size of the reference set increases,  the number of reference-residual pairs grows combinatorially. For e.g., when the reference set is the entire dataset D, there are |D| choose 2 pairs, making it impractical to explore all pairs within a fixed number of training iterations. This results in insufficient sampling of reference-residual pairs, increasing the risk that anchored training may overlook the reference and make predictions based solely on the residuals leading to non-generalizable shortcuts. This is problematic as a sample should not be identifiable without knowing the reference.  \n\n**2. Alleviating the Reference Set Size Problem**\n\nOne possible way of alleviating this problem is by reducing the reference set size. However, this reduces the diversity of the reference-residual pairs exposed during training and can lead to a poor solution. While the issue of diversity can be combated with large reference set sizes, increasing the number of epochs alone does not solve the problem as there exists a combinatorially large number of reference-residual pairs which cannot be practically explored, and the model will still be vulnerable to shortcuts.  Moreover, modifying the number of training epochs results in non-trivial modifications in the training hyper-parameters (e.g., learning rate schedules) and can lead to poorly convergent models if the hyper-parameters are chosen incorrectly.  Hence, we propose a reference masking regularizer for anchored training, that helps mitigate shortcut decision rules while also being computationally efficient.  \n\n**3. Reference Set Selection Strategy**\n\nFor all experiments in Section 4, we utilize the entire training dataset as the reference set and train both the original and the proposed anchored models. During inference, we randomly select a single reference from the reference set and perform evaluation on the different test datasets.  We will better clarify this in the final version of the paper. \n\n**4. Impact/Novelty**\n\nAnchoring is a framework that is agnostic to any training strategy, domain or application and can be wrapped along with other strategies (data augmentation, loss functions, regularizers, ensembling) that help improve model performance. While this is attractive, our paper identifies the shortcomings of the existing protocol and deals with a fundamental problem of how to train and make predictions with anchored models in practice. We develop a novel reference masking protocol for training anchored models that can significantly improve overall model generalization. We demonstrate significant quantitative performance improvements over standard and original anchored training protocol across different datasets, tasks and architectures. Particularly, we find that our proposed algorithm leads to a wider and flatter optima corresponding to superior solutions (Fig. 4), and can be used on top of augmentation strategies (Fig. 5a) and better handle training label noise (Fig. 5b). We systematically establish the empirical efficacy of our approach on OOD generalization and model safety tasks ranging from calibration, anomaly rejection as well as task adaptation and domain generalization. We expect to foster interesting research directions with anchoring and even impact applications in different domains (e.g., text, graphs etc.) where generalization and model safety are paramount concerns. \n\n**5. Formatting Issues and Typos**\n\nWe will make the changes to the tables and figures in the final version of the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates anchoring-based training of vision models. It indicates that reference diversity is important for anchoring training to achieve generalization, and existing methods tend to learn spurious shortcuts  due to insufficient sampling. It further proposes a masking strategy to improve performance and efficiency. Extensive experiments demonstrate the effectiveness of the proposed approach.

Four experts reviewed this paper and all recommended acceptance. The reviewers liked that the work presents an important limitation to existing anchoring techniques that was unknown to the community, and proposed a simple and effective method to improve both OOD and safty in anchoring training.
Moreover, the discussion is comprehensive and provides useful guidelines for how to use the proposed strategy to achieve maximum performance. The paper is very well-written and easy to follow. Based on the reviewers' feedback, the decision is to recommend the paper for acceptance.
Decision: Accept (spotlight)