Predicting the Performance of Foundation Models via Agreement-on-the-Line
Authors: Rahul Saxena
Taeyoun Kim
Aman Mehra
Christina Baek
J Zico Kolter
Aditi Raghunathan
Venue: NeurIPS 2024 poster (2024)
Keywords: robustness
OOD performance estimation
foundation models
agreement-on-the-line
Primary area: safety_in_machine_learning
TL;DR: We find methods for leveraging agreement-on-the-line for estimating of the performance of finetuned foundation models under distribution shift.

Abstract: Estimating the out-of-distribution performance in regimes where labels are scarce is critical to safely deploy foundation models. Recently, it was shown that ensembles of neural networks observe the phenomena "agreement-on-the-line", which can be leveraged to reliably predict OOD performance without labels. However, in contrast to classical neural networks that are trained on in-distribution data from scratch for numerous epochs, foundation models undergo minimal finetuning from heavily pretrained weights, which may reduce the ensemble diversity needed to observe agreement-on-the-line. In our work, we demonstrate that when lightly finetuning multiple runs from a $\textit{single}$ foundation model, the choice of randomness during training (linear head initialization, data ordering, and data subsetting) can lead to drastically different levels of agreement-on-the-line in the resulting ensemble. Surprisingly, only random head initialization is able to reliably induce agreement-on-the-line in finetuned foundation models across vision and language benchmarks. Second, we demonstrate that ensembles of $\textit{multiple}$ foundation models pretrained on different datasets but finetuned on the same task can also show agreement-on-the-line. In total, by careful construction of a diverse ensemble, we can utilize agreement-on-the-line-based methods to predict the OOD performance of foundation models with high precision.

Predicting the Performance of Foundation Models via Agreementoon-te--Line
Rahul Saxena"l
Taeyoun Kim Aman Mehra*1
Christina Baek
Zico Kolter1.2
Aditi Raghunathan'
Carnegie Mellon University Bosch Center for A1 (reaxena2, taeyoun3, amanmehr, xbaek, zkolter, raditi)@cs. cmu edu
Abstract
Estimating the out-of- ditssriibbutinn performance in regimes where labels are scarce is critical to safely deploy foundation models. Recently, was shown that ensembles of neural networks observe the phenomena "agreement-on-the-llne leveraged to reliably predict OOD performance without labels. However, contrast to classical neural networks that are trained on in-distribution data from scratch for numerous epochs, foundation models undergo minimal finetuning from heavily pretrained weights, which may reduce the ensemble diversity needed to observe agreement-on- gureement-on-the-line In our work, we demonstrate that when lightly finetuning multiple runs from single foundation model, the choice of random- ness during training (linear head initialization, data ordering, and data subsetting) can lead to drastically different levels of agreement-on the-line in the resulting ensemble. Surprisingly, only random head initialization is able reliably induce prreement-on-thee tee--iheeeee in finetuned foundation models across vision and language benchmarks. Second, we demonstrate that ensembles multiple foundation mod- els pretrained on different datasets but finetuned on the same task can also show agreement-on -ot-eentt----h--lieeee-- In total, by careful construction of a diverse ensemble, we can utilize -ot-eentt----e---ieeeee of foundation models with high precision.
which can
methods to predict the OOD performance
Introduction
Foundation models (FM). or large models first pretrained on open world data then finetuned or prompted for specific downstream task, have proven to be powerful solutions for many common machine learning problems. A notable trait about FMs is that they are far more robust distribution shift than other deep learning approaches across image and language benchmarks, they suffer smaller performance degradation on out-of-distribution (OOD) data, that may vary substantially 58 14] From clinical docision-making in different hospitals to navigating robots through unseen terrains, FMs increasingly utilized for tasks prone to distribution shift. However, evaluating these models in OOD settings remains difficult: many cases, acquiring labels for OOD data is costly and inefficient, while unlabled OOD data is much easier to collect. Although the field has explored other means for estimating OOD accuracy without labeled data, they are not ideal for FMs. reliable FM performance estimator has the following desirable properties. First, the method must be computationally efficient to account for FMs' large model size. Second. FMS are leveraged for many different tasks (c.g., classification. question inestion-annswring answering, regression), so the method should also be versatile across tasks. Third, as we will see, methods for finetuned FMS may require different model assumptions from neural networks trained from scratch.
from the in-distribution (ID) finetuning data [43 42 7
60
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

CIFAR1OC (Pixelate)
CIFAR1OC (Pixelate)
CIFAR10C (Pixelate)
90
Accuracy
Accuracy
Accuracy
Agreement
Agreement
Agreement
70
000
50
30
20-
10
ID
ID
SNU
SNLI
SNLI
Accuracy
Accuracy
Accuracy
Agreement
Agreement
Agreementt
00D
00D
50
30
30
30
SQuAD-Shifts (Amazon)
SQuAD- Shifts (Amazon)
SQuAD-Shifts (Amazon)
90
90
F1 Score
F1 Score
F1 Score
Agreement
Agreement
Agneement
000
000
30
30
10
30
50
70
90
10
30
50
70 90 10
30
50
70
50
ID
ID
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure I: The ID vs OOD lines for accuracy (orange) and agreement (blue) for various datasets and fine-tuned ensembles. Each blue dot corresponds to member of the ensemble and represents the ID (x) and OOD (y) accuracy. Each orange dot corresponds to pair of these members and represents the ID (x) and OOD (y) agreement. From CIFAR10 to CIFAR1OC "Pixelate" in linear probed CLIP, MNLI to SNLI in full fine-tuned OPT, and SQuAD to SQuAD- -hifts "Amazon" in full fine-tuned GPT2, observe that randomly initializing the head the diversity source for generating ensembles (columns) shows the closest agreement linear fit to accuracy.
Recently, 22 proposed promising method for estimating the OOD accuracy of deep networks using the agreement between pairs of these classifiers (i.c., how often two classifiers make the same prediction). For distribution shifts where models observe strong linear correlation in ID versus OOD accuracy a common phenomenon in vision and language benchmarks [39 [ strong linear correlation also holds for ID versus OOD agreement with extremely similar slopes and intercepts. These effects are referred to as accuracy-on-the-ling (ACL) and agreement-on-the-iene (AGL) respectively, and together they provide simple method for estimating OOD accuracy via unlabeled data alone. Namely, without any OOD labels, we can instead measure the linear of ID versus OOD agreement as proxy for the linear fit of ID versus OOD accuracy. With this linear fit, we can verify whether ACL holds by using the correlation strength agreement's linear trend and estimating each model's OOD accuracy by linearly transforming ID accuracy. This simple approach has shown to reliably predict the OOD accuracy models within a few percentage points across classification and muestion-answering tasks.

---

Unfortunately, while the method has several practical advantages, it is unclear whether finetuned FMs also observe the necessary AGL phenomena. Intuitively, prerequisite to observing AGL is a diverse ensemble of classifiers. Since OOD accuracy falls below ID accuracy. if the linear trend match that of accuracy, models must also agree much less OOD
in ID versus OOD agreement than ID. For this to happen, errors between any two models must be sufficiently decorrelated. 2 observes AGL in ensembles of neural networks trained for hundreds of epochs from scratch where is conceivable that the stochasticity between training runs leads to large divergences in the weight space, and corresponding models have diverse OOD predictions. However, the case of finetuned FMs, models are much closer in the weight space. FMs are often either linear probed over the same pretrained weights full finetuned for a few epochs with a small learning rate and intuitively, such light finetuning may lead to models that "revert back" to their pretrained behavior to make highly correlated predictions OOD.
This raises the question: can we enforce AGL in this paradigm of lightly finetuning heavily pretrained models? In this work, conduct an extensive study across several modalities, e-g., CL.IP-based image classification and LLLM-based question-answeringg and linear probing, to understand when AGL holds for finetuned FMs. We first investigate whether AGL appears in an ensemble of finetuned models from a single base FM. To collect a deep ensemble, the following sources of diversity can be injected into the finetuning process: 1) random initialization of the linear head; 2) random data ordering; and 3) random data subsetting. We find that not every ID data manifests in sufficient diversity OOD, breaking the matching linear fits in ID versus OOD accuracy and agreement. Interestingly, finetuning models from different random initializations of the linear head consistently induces AGL across benchmarks. In contrast, neural networks trained from scratch observe AGL irrespective to these diversity sources.
and training regiments, e.g.. full finetuning
source of diversity during fine-tuning
Second, we show that finetuned models from multiple different base FMs can be leveraged for AGI.- based performance estimation. As base FMs can be pretrained with different datasets, architectures, and training regiments, the linear trends in ID versus OOD accuracy and agreement may break altogether in such ensembles. Indeed previous works indicate that on vision tasks, FMs pretrained on different image corpora can have different levels of OOD robustness for the same ID performance [7 43 551 On the contrary, we find that on language tasks, FMs pretrained on different text corpora observe both AGL and ACL across question-ant guestion-answering answering and text classification tasks.
In total, we develop simple techniques for applying AGL-based performance estimation methods to predict the OOD performance of foundation models. We demonstrate that the AGL phenomenon is not limited to ensembles of neural networks trained from scratch. By simply finetuning FMs from random initializations of the linear head, we can observe the phenomena in FMs across a wide variety of tasks (classification, question-answeringg and modalities (vision, language) and training procedures (linear probing, full finetuning). We find that AGL. is the only method to accurately estimate the performance of finetuned FMs across all tasks, surpassing other performance estimation baselines by a significant margin as large as 20% mean absolute percentage error.
Background and related work
2.1 Setup
We are interested in evaluating models that map an input x E X to a discrete output y Y. In particular, we finetune foundation models. For a base model B, let f(B) denote finetuned version of B. In this work, we consider a variety of foundation models: GPT2 42 OPT 65 Llama2 [3].
BERT 14, and CLIP [3
Finetuning strategies. We have access to labeled data from some distribution DDd that we use for obtaining (BB from B. In this work, we consider the following standard finetuning procedures.
1. Linear probing (LP):: Given features from the base model Ba, we train a linear head v such that the final classifier maps the score Bx(x) to a predicted class. We randomly initialize u and update via gradient steps on a suitable loss function. The base model parameters remain frozen. We refer to v as either a linear probe (classification), or span prediction head (question- (question-answerrng answering) depending on the task.
3

---

2. Full finetuning (FFT); We update all parameters of the backbone By and the linear head using small learning rate. When infeasible to update all parameters, we perform low-rank adaptation (LoRA) [25] to reduce the number of trainable parameters while still effectively updating the feature extractor Bg. In this work, we do not distinguish between LoRA and FFT as they conceptually achieve the same effect, and seem to show similar empirical trends in our studies.
OOD performance estimation. Given access labeled validation set from PiD and unlabeled samples from a related but different distribution DOOD, our goal is to estimate performance on DOOD.
We consider the standard performance metrics for various tasks: Accuracy fo-1 Y + [0, 1] for
classification, and Exact Match fEM Y + [o, 1] and Macro-averaged Fl score 41 Y + [0, 1] for question-answering We use to denote the appropriate metric in the context.
2.2 Background on OOD accuracy estimation
There rich literature on OOD performance estimation for deep networks, with variety of proposed approaches. Initial works focused on upper bounding the degree of distribution shift through data and/or model dependent metrics, e.g., uniform convergence bounds using H-diiergence [1] 30) However, these bounds tend to be loose for deep networks [39] The following works try to estimate the performance exactly.
leverage the model's confidence to predict the OOD perfor- mance. Since deep models typically overconfident, these models are first calibrated in-distribution by temperature scaling. Similar methods are uncertainty quantification works that directly calibrate models under distribution shift 662 67 41) Confidence based methods commonly utilized in practice, and favorable for foundation models as they are computationally light and model-agnostic. However, they often fail for large shifts [19] and are often well-defined for accuracy but not other common metrics like F1 score. These can be limiting factors for foundation models which are applied to broad array of tasks. Still, they are the most common estimation methods, we utilize them as the baselines in our work.
For classification, [23 22
16
also measure model behavior on known auxiliary tasks to understand model behavior under the distribution shift at hand. However, these approaches tend be overfit to specific datasets or modalities. Similar to AGI., there are prediction methods that utilize information from ensembles. Oftentimes separate "reference" ensemble is trained on some objective to predict the performance of "target" model [10 63 3 These methods have higher computational cost than AGL. Although AGL also requires at least models to compute agreement, these models only undergo generic finetuning. Thus, it is better suited approach for evaluating foundation models, especially if off off-the-shelf finetuned models are readily available, e.g., from Huggingface (see Section 4).
49 12 13
Overall, there is growing attention towards understanding the safety and reliability of foundation models. To understand the effective robustness of FMs under distribution shift, recent works focus on studying the baccurac--on-the-lin benchmarks that expose different failure modes of large models [36 54 However, unsupervised OOD performance estimation underexplored in this modern setting, in terms of new methods and the transferability of old methods large pretrained models.
phenomena (39] (details in next subsection) and designing
2.3 Accuracy and agreement on the line
We are interested in adapting the method "agreemeent-on-the-iiie (AGL) 2 for OOD estimation as it obtains ofate-of-the-arr performance estimation across several distribution shifts. AGL is based on an earlier observation called "accuracy-on-the-iin
(ACL) across common distribution shift benchmarks, there is a strong linear correlation between the ID and OOD performance of models 51 38) ACL can also be observed in FMs for image classification, e.g., COFARIOC ImageNet [4). FMOW-WILDS [88. and question-answering. e.g., SQuAD-Shifts (38]
33
47
61
122.
However, ACL does always hold, e.g., WIID 39) and SearchQA M.
While ACL is striking phenomenon, it does not immediately provide a practical method to estimate OOD perfornance-compurigg the linear fit of ID versus OOD accuracy requires labeled samples from DOOD- Alternatively, we can estimate this linear trend exactly using only the agreement between neural networks (2). Formally, given pair of models fi and f2 that map inputs to labels, accuracy

---

and agreement is defined as
Acc(fi) Exx.y- Agr(Ji, f2) = En
(I)
where l is the appropriate performance metric of interest. While accuracy requires access to the ground truth labels agreement only requires access to unlabeled data and a pair of models. [2] observes that when ID versus OOD accuracy is strongly linearly correlated between neural networks, i.e., ACL, then the ID versus OOD agreement of pairs of these models also observe strong linear correlation with the same linear slope and bias. Furthermore, when accuracies do show linear correlation, agreements also do not. This coupled phenomenon is dubbed aggreement-on-th--leee (AGL).
To use AGL for OOD performance estimation,, one may obtain the slope and bias of the agreement line with unlabeled data, and then estimate the OOD performance by linearly transforming the ID validation performance. Specifically, with a collection of models F = {f1, f2. .... fn), AGL suggests that ID versus OOD accuracy observe strong linear correlation if and only if ID versus OOD agreement observes a strong linear correlation and when they do, the slopes and biases match: ff, fa E JF where i *
Φ (AccooD((()) = a. =a- (=-((Acc((Fi)) +
$
(2)
(Agroon = a 1 f)))+ +
Φ- is the probit transform used induce better linear fit as used in [2]39] Provided access to Ag fy) vi we can estimate AccooD(ft) for all fi E F. We refer the reader [2] for formal AGL-based performance estimation algorithms (ALine-S and ALine-D), which we also provide in Appendix A.I.I
Accm(fi)) (ft. f,).
3 Predicting OOD performance: single base foundation model
We first evaluate whether AGL appears in ensemble of multiple finetuned of single base foundation model. This would enable precise OOD performance estimates for each ensemble member. practitioner may naively gather finetuned ensemble by training a couple seeds or hyperparameters. However, an overriding concern is that even with some randomness in the finetuning process, linear probing light full-finetuning over the same base model may lead to solutions with very correlated predictions. We extensively evaluate the following methods of introducing diversity into the finetuning process to what approach (if any) lead to AGL.
with different
1. Random linear heads We initialize the last layer of the network (i.e., the linear head) randomly, instead of via some zero-shot or pre-specified manner.
2. Data ordering We present the same training data to each model but shuffle the order of the data, i.c., model observes different minibatches.
3. Data subsetting We i.i.d. sample p% subset of the data to train over. In the main body, we report models trained independently sampled 10% of the training data, other proportions of 30% and 50% are reported in Appendix A.4
We perturb one source of diversity at time and study whether AGL occurs in each resulting model ensemble. For each setting, we also vary the number of training epochs to collect models with different ID performance, which is necessary to obtain a meaningful linear correlation in accuracy. The additional randomness induced by different training epochs does not affect the observations we make. We use at most four A6000's s for all experiments except for linear probing where we use one RTX 8000.
3.1 VLM-based Image Classification
We first investigate the effect of diversity source on AGL. behavior for vision benchmarks. For image classification, common pipeline is finetune over CLIP [43] pretrained foundation model.
5

---

CLIP Linear Probing We finetune over OpenCLIP ViT-B/32 model trained on LAION-2B [26]. Given its well-established zero-shot capabilities, a popular method of finetuning CLIP simply employ linear probing on top of the CLIP representation. We take particular interest in evaluat- ing the OOD performance of an ensemble of linear models trained on top of frozen base model representations.
Datasets We evaluate ensembles on synthetic corruptions (CIFARIOC, CIFARIOOC, ImageNetC), dataset replication shifts (CIFAR10.1 ImageNetV2), style shifts (OfficeHome). geographical and temporal shifts JFMoW-WILDS, (Camelyon 1oan17-WWIII. WILDS). iWildCam- WILDS exhibits weak ACI and Camelyon 7-WIIDS doesn't 't exhibit any ACI 39 We test on iWildCam- WI and Camelyon17-WI.J negative condition, i.e., when the linear correlation does not exist in ID versus OOD accuracy, it also does not exist in agreement.
WiildCam-WILDS. and interlaboratory shifts in medicine
to verify AGL's
Table I: We evaluate models on the following distribution shift benchmarks.
ID
0OD
CI
29
CIFAR 22]. CO.AR
A5
CIFAR100
20
CIFARI00C
ImageNet
43
ImageNetC (22], CIFARIO.1
HMoW ID
HMoW ODD 28
:WildCam ID
28
iWildCam OOD
28
Camelyon
Camelyon!
OfficcHome [5c
AlI OOD) pairings of domains Art, ClipArt, Product, Real World
MNLI 59
MNLI- Mi-smmitttteedd 59 SNLI 6
SQuAD
44
SQiADsShifts [38
CIFARL0OC (impulse Noise)
ImageNetC (Contrast)
fMoi WILDS (OOD)
Camelyon17 WILDS (OOD)
Accrway
Agreement
Agremment
Agiccmant
Agraement
73-
000
C0O
33
Figure 2: In ensembles with diverse random initializations, ACL AGL holds across benchmarks in linear probed CLIP models. Similar to [2], neither ACL nor AGL holds for the Camelyon17-Wi WI-LLLSS
Results Across vision benchmarks, linear probed CLIP models observe ACL, i.c. there is a strong linear correlation in the ID versus OOD performance. Similarly, the same datasets, we can observe a corresponding strong linear correlation in agreement across ensembles injected with diversity in linear head initialization, data ordering. and data subsetting (Figures] Dp. However, find that only ensembles with diverse initialization leads to AGL where the linear trend of agreement and accuracy have matching slope and bias (Figure 2 See Appendix A.3.2 for results other datasets. In model ensembles obtained by data ordering and data subsetting, we observe a consistent trend where the agreement trend observe much higher slope close the diagonal y = line. These results are not specific to linear probing alone. In full finetuned CLIP models, we also observe that random linear heads induce the most reliable AGL behavior (See Appendix (A.)). Note that this setting is still notably different from (2) where models are heavily trained for tens to hundreds of epochs often with large learning rate, which causes AGL behavior to be more robust to the source of diversity used to induce the ensemble (See Appendix A.3.44 A.3.4).
3.2 LLM-based Question-Ansswrring and Text Classification
We conduct similar systematic investigation of AGL in finetuned runs of a single base language model. Similar to CLIP linear probing, we find that AGL cannot be observed without random head initialization in language models evaluated on text classification and extractive question-answering tasks. While we mostly focus on tasks that require a linear head during finetuning, we also conduct a diversity study generative tasks where the base model is finetuned directly in Appendix A.6.

---

Table 2: ALine-D MAPE (%) of different sources of diversity for CLIP linear probing and GPT2- Medium/OPT-12255 full finetuning. We average the score over all corruptions for CIFARIOC.
Source of Diversity
CIFARIOC SQuAD-Shifts Amazon SQuAD-Shifts Reddit
SNLI
Random Linear Heads
14.64
6.34
3,48
11.70
Data Ordering
37.01
10.30
9.59
15,40
Data Subsetting
35.85
16.21
13.94
15.50
Full Finetuned Language Models We evaluate over a collection of 450 full finetuned runs of several base FMs: GPT2-Medium [42 and OPT-125M (65). Models are full finetuned for up to 20 epochs with a small learning rate (S le 9. Hyperparameters specifies can be found in Appendix
A.2 We do not conduct linear probing study for question-answering as it leads to poorly performing models. For text classification, we also conduct a linear probing study Appendix A.3
Datasets We test models text classification shift from MNLI 55 in the GLUE benchmark 57 to MNLJ-Mismatched [59] and SNLI [6] We also evaluate extractive question-answering models on the shift from SQuAD v1.1 44 to SQuAD-Shifts (8).
Results We evaluate models on accuracy for text classification and Fl score for question-answering. Similar our findings in CLIP, in both text classification and question-answering benchmarks, ensembles of full finetuned LLMs observe AGL when models are trained from different randomly initialized linear or span heads while data ordering and data subsetting observe an agreement trend closer to the diagonal = x line (Figure and Appendix A.5). We note that with full finetuning, the differences in AGL behavior between diversity sources are not as stark as with linearly probed models. In some sense, how model diversity is achieved becomes increasingly less important for observing AGL as the base model parameters also diverge, with ensembles of models heavily trained from scratch at the extreme (2).
3.3 Summary and Implications
Across image and language modalities, we demonstrate that ensembles of finetuned FMs can also observe srreement-on-the-line similar heavily trained CNN's (2]. In both domains and regardless of the fine-tuning strategy, e.g. FFT and LP, or metric, e.g. F1 and Accuracy, employed, the diversity induced via random head initialization yields AGI., while the diversity induced via data reordering or data subsetting does not. This phenomenon can be observed across hyperparameters [Appendix[A.] and different PEFT methods (Appendix (A.)). With single heavily pre-trained base FM, one may think that light finetuning leads to downstream models with highly correlated behavior under distribution shift. However, simply randomly initializing the linear head alone induces sufficiently decorrelated models for observing AGL. The diversity in the ensemble becomes important when predicting the OOD performance of models using downstream AGL-based methods. In Table 2 we show that AGL-based methods can only accurately predict the OOD performance of models in ensembles with diverse initialization, and cannot with data subsetting or ordering.
Furthermore, our findings contrast previous work that suggest AGL is neural-network specific phenomenon (2 (1). unlike ACI which is model agnostic (39. Specifically, 2 report that linear models trained on top of the flattened CIFARIO images do not observe AGL. However, we find that, on top of CLIP features, linear models can exhibit AGL with random initialization. Previous work on the Generalization Disagreement Equality 27 contend that data subsetting leads to the most diversity in model predictions for deep ensembles (ncural networks, random forests). Specifically, in-distribution, the agreement rate between pairs models was shown to equal their expected accuracy in ensembles obtained by data subsetting, while those that vary random initialization has slightly higher agreement (27 40). On the other hand, in our problem setting of out-of-diittribution datasets on FMs, we found that ensembles induced by different random initialization achieves AGL, while data ordering or subsetting cannot. Our setting is different from previous literature in two ways: (1) AGL studies the OOD agreement rate relative to their ID agreement, in contrast to the GDE phenomenon which only regards the models' ID agreement. We hypothesize that random initialization is much more important for observing the right levels of OOD agreement. (2) Models
7

---

SQUAD-Sh fit (Amazon)
SQUAD-Shifts York
MNLI Mismatched
ID
ID SNLI
SQuAD-Shifts (Reddit)
SQuAD-Shifts (New Wiki)
000
D00
34
30
59
10
53
96
85
Agreement
Liama
GPT
Figure 3: AGL can be observed between models finetuned from different base models (Llama, GPT, OPT) for the Fl score for question-answert shift (SQuAD to SQuAD-Shifts) and accuracy for text classification (MNLI-Matched to ENLI--ismatched and SNLI). SQuAD-Shifts New Wiki, SQuAD-Shifts NYT, and MNLI Mismatched show little drop in OOD performance because the distribution shift is small compared to the corresponding II) dataset. Nonetheless, we observe that AGL holds regardless the degree of distribution shift.
are only lightly fine-tuned or linearly probed, unlike deep ensembles trained from scratch. Diversity sources may behave differently in this circumstance.
4 Predicting OOD performance: multiple foundation models
Alternatively, consider ensembling multiple base foundation models. First, ACL may not hold
because the base models are heavily pretrained on different data corpuses. This may cause respective downstream models to have diffferent ID versus OOD accuracy trends or "effective robustness" [17].
On vision tasks, for example, linear probing over CLIP, EfficientNet [50). ViT [15), and BYOL [20]
observe varying robustness trends (43). Second, even when ACL does hold, it is unclear whether the ensembles will also observe AGL. Here the problem is different from the single base model setting: any pair of foundation models finetuned from different base models may agree too little, or OOD agreement rate may vary across model pairs depending on similarity of the pretraining corpus, breaking the linear correlation of agreement entirely. Yet, observe that for language models and tasks, ensembles of finetuned FMs from wide range of base models observe both ACL and AGL.
Models We finetune models from OPT-125M, OPT-350M, OPT-1.3B 65 GPT2, GPT2-Mediume
GPT2-Large, GPT2-XL 42). GPT-Neo-135M (5). Llama2-7B (53] Alpaca-7B [52], and Vicuna-7B
[ We fully finetune OPT and GPT models and LoRA finetune Llama, Alpaca, and Vieuna. These models are pretrained on different mixtures of BookCorpus (66), Stories (55), PILE (18), CCNews v2 corpus, and PushShift.o Reddit 3 Alpaca and Vicuna are instruction
instruction-finetuned finetuned over Llama2.
4.1 Results
We investigate the AGL behavior of an ensemble of foundation models finetuned from diverse base models in Figure for question-an -anwwernn.....rigg...... First note that base LLM models pretrained on different text corpora lead to finetuned models that lie on the same linear trend in accuracy. Unlike the different accuracy trends observed by different vision foundation models (42. we suspect that the pretraining datascts for the language models in study observe much more homogeneity. Second, the ID

---

Table The MAPE (%) of predicting OOD performance using AGL-based ALine and other baseline methods. We collect diverse ensemble by randomizing the linear initialization and including multiple base models. We filter out shifts with low correlation in agreement.
OOD Dataset
ALine-D
ALine-S
Naive Agr
ATC
AC
DOC-Feat
CIFARIOG
5,44
4.73
17.39
6.90
11.49
11.91
CI.AR10.1 v6 CIFAR1OOC*
1.95
1.99
16.95
2.60
4.93
5.36
7.17
6.79
17.66
8.18
17.58
14,96
ImageNetC*
15,03
14.17
32.27
15.90
22.83
13.42
ImageNetV2 MatchFreq
8.44
8.43
22.21
3.02
15.53
8,43
MMW--WI.DS
14.26
7.29
141.31
12.17
19.87
8.76
OfficeHome-Ar
17.78
13.94
40.60
38.52
19.68
44.40
PiiceHomee-CipApe
14.04
11.70
33.44
33.81
14.97
28.77
OfficeHome-Produe
14.64
11.78
39.88
84.18
63.05
75.44
OfficeHome- Real
12.65
10.18
36.20
27.72
21.85
28.85
SQuAD- Reddit
3.61
3.48
26.56
19.06
30,94
9.18
SQuAD- D-iif-Siifts Amazon
3.61
4.93
26.46
24.35
34.93
7.31
SQuAD-Shifts NYT
1.64
1.75
23.46
4.01
25.96
2.80
SQuAD-Shifts New Wiki
6.33
6.58
25.24
5.18
25.96
7.50
MNLI Mismatched
0.55
0.41
0.55
12,00
0.63
0,51
SNLI
2.90
2.10
2.50
6.30
3,80
8.40
versus OOD agreement between pairs of models in this ensemble, including those between different base foundation models, is also strongly correlated and the slope and intercept closely matches that of accuracy. In other words, ensembles of different base models also observe AGL without any special regularization for ensemble diversity. The same holds for generative QA tasks (Appendix (A...)).
5 Estimating OOD Accuracy using AGL in Diverse Ensembles
By constructing a diverse ensemble of foundation models, we can leverage AGL to extract precise estimates of model performances under distribution shift. We construct ensembles by collecting models trained from randomly-nnitialized heads (Section 3) and different base models (Section 4). For image classification, our model collection consists just linear models over CLIP representations. For text classification and question-answering, we include GPT, OPT, and Llama models individually
finetuned from differently initialized heads. In Table 3 we compare the Mean Absolute Percentage
Error (MAPE) of AGL--based prediction algorithms, ALine-S and ALine-D
2 other baselines.
We compare against confidence based methods ATC 19], AC [24 and DOC-Feat [21] and Naive Agreement which directly uses agreement between model pairs 35 For confidence based meth- ods, we first temperature scale the models using ID validation data, and pick the lower error rate from the estimations obtained with and without temperature scaling. However, there are several limitations when naively applying confidence baselines to estimate performance on question-answeriin estimate classification accuracy. First, there is no easy analogous formulation of confidence baselines for the Fl score, so we estimate the exact-match score instead for fair comparison. On the other hand, AGL predict performance question-answering is joint classification task where models predict both the start and end token index of the answer span in the context. More details for how calibrate baselines for this setting is provided in Appendix 1.1.2
as they
metrics accuracy, F1, and exact-match. Second, extractive
Because ALine-S and ALine-D only provide estimation guarantees where the coefficient of deter- mination 122 of the linear in agreement is strong (2)), we filter out datasets with low R² < 0.95. Camelyon-WILDD
These shifts include WWlldCm--WILDS CIFARI00C and ImageNetC.
and few corruptions in CIFAR 10C, evaluate ALine-S/D for these failure cases Appendix A. Across datasets with a high Rta in agreement, ALine-S and ALine-D provide precise OOD perfor- mance estimates finetuned FMs, surpassing other baselines by a large margin. This is noteworthy, especially for shifts where the agreement line is significantly off of this method. Furthermore, they perform better on the next best confidence method achieving large as 20% higher error.
further lending to the utility task SQuAD, with the

---

6 Limitations
Estimating the tuttof--iitributionn performance of foundation models has rapidly grown in importance, especially as these models are increasingly deployed in real-world use cases. Our work focuses on promising method enable deployers to reduce the harm of machine learning systems when they encounter OOD inputs. However, deployers should be careful to not use AGL as the only signal for OOD performance. The correlation between agreement and accuracy not guaranteed hold for all distribution shifts, so other metrics should additionally be used to monitor model performance. In particular for foundation models, we observe that careful choices during tinettunig is required to observe AGL. In fact, if different pretrained checkpoints are evaluated zero-shot, without any fine- tuning, AGL not able to reliably predict the performance of FMs (Appendix AA3333) Furthermore, while we studied AGL closely for a wide array of classification/QA benchmarks, there remains other important downstream tasks such as long-form generation that we leave for future study. We also do not provide any theoretical guarantees to back our empirical findings, such as the importance of random initialization, which we leave for future work.
Conclusion
We develop methods for extending AGL to foundation models enable OOD performance prediction in this emerging paradigm. We find that utilizing AGL for performance estimation requires a careful tuning of ensemble diversity. Unlike the original paradigm of AGL, where models observed tens hundreds of epochs of training on the ID dataset, we find that randomness in specific optimization choices, especially linear head initialization, is crucial for foundation models. In fact, in contrast to (2). we find that linear models can also observe AGL, specifically in the CLIP representation space, suggesting that AGL may not be neural network specific phenomena. Our conclusion sheds light on the robustness of foundation models. First, our experiments show that light finetuming alone can corrupt models to have diverse behaviors. Next, in contrast to vision models, where previous works show different forms of pretraining lead to different slopes in the linear correlations 43), find that all the language models evaluate, e.g.. OPT, GPT2, GPT2-Neo, Alpaca, Llama, and Vicuna lie on the same accuracy line. This is particularly intriguing because it goes against the common wisdom that the pretraining data influences the models' "effective robustness", We leave these questions for future analysis.
AGL also
References
[1] A. Awadalla, M. Wortsman, G. Ilharco, S. Min, I. Magnusson, H. Hajishirzi, and L. Schmidt, Exploring the landscape of distributional robustness for question answering models. arXiv preprint arXiv:2210. 125177, 2022.
[2] C. Back, Y. Jiang, A. Raghunathan, and J. Z. Kolter. Agreement-onnthe-linne Predicting the performance of neural networks under distribution shift. Advances in Neural Information Processing Systems, 35:19274-19289, 2022.
[3] J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, and J. Blackburn. The pushshift red- dit dataset. In Proceedings of the international AAAI conference on web and social media, volume 14, pages 830-839, 2020.
[4] S. Ben-David, J. Blitzer, K. Crammer, and Pereira. Analysis of representations for domain adaptation. Advances in neural information processing systems, 19, 2006.
[5] S. Black, L. Gao, P. Wang, C. Leahy, and S. Biderman. GPT-Neo: Large Scale Autoregres- sive Language Modeling with Mesh- Beesh-Tensorloow, 2021. URL htttt::////oii. org/10. 5281/ (5tod1552977555
[6] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning. large annotated corpus for learning natural language inference. arXiv preprint (eXXiv1508,,05333 2015.
[7] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, Askell, et al. Language models are few-shot learners. Advances neural information processing systems, 19:18 2020.
10

---

[8] J. Chen. F. Liu, B. Avci, X. Wu, Y. Liang, and S. Jha. Detecting enrors and estimating accuracy on unlabeled data with self self-training ensembles. arXiv preprint arXiv:2106. 15728, 2021.
[9] W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng. S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL http::///aaaasrrrr/////////22220
[10] C.-Y. Chuang, A. Torralba, and S. Jegelka. Estimating generalization under distribution shifts via domain-invariant representations. arXiv preprint arXiv:2007.03511, 2020.
[11] C. Cortes, Y. Mansour, and M. Mohri. Learning bounds for importance weighting. Advances in neural information processing systems, 23, 2010.
[12] W. Deng and L. Zheng. Are labels always necessary for classifier accuracy evaluation? In IEEE/CVF Conference on Computer Vision and Pattem Recognition (CVPR), pages 15064- 15073. IEEE Computer Society, 2021. doi: 10. 10.1109/CVPR44447.0 202.00088..
[13] W. Deng. S. Gould, and L. Zheng. What does rotation prediction tell about classifier accuracy under varying testing environments? arXiv preprint vrXiv:2106.0591, 2021.
[14] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1xXiv:181000480) 2018.
[15] A. Dosovitskiy, L. Beyer. A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[16] H. Elsahar and M. Gallé. To annotate or not? predicting performance drop under domain shift. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)) pages 2163 2173,, 2019.
[17] A. Fang, G. Ilharco. M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt. Data determines distributional robustness in contrastive language image pre-training (clip). In International Conference on Machine Learning, pages 6216-6234, PMLR, 2022.
[18] L. Gao, S. Biderman, S. Black, L. Golding, T. Hoppe, C. Foster, J. Phang, H. He, A. Thite, N. Nabeshima, et al. The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027 2020.
[19] S. Garg, Balakrishnan, Z. C. Lipton, B. Neyshabur, and H. Sedghi. Leveraging unlabeled data to predict out-of performance. International Conference on Learning Repre- sentations, 2022.
[20] J.-B. Grill, F. Strub, Altché, C. Tallec, P. Richemond, E. Buchatskaya, C. Doersch, B. Avila Pires, 7. Guo, M. Gheshlaghi Azar, et al. Bootstrap your own latent-a new ap- proach to self-supervised learning. Advances in neural information processing systems, 33: 21271-21284, 2020.
[21] D. Guillory, V. Shankar, S. Ebrahimi, Darrell, and L. Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF international conference computer vision, pages 1134-1144, 2021.
[22] D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR, 2019.
[23] D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distributioo examples in neural networks. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
[24] D. Hendrycks and K. Gimpel. baseline for detecting misclassified and distr--iittintttio examples in neural networks. International Conference Learning Representattinss 2017.
11

---

[25] E. J. Hu. Y. Shen, P. Wallis, Z. Allen-Zhu, Zhu, Y. Li, S. Wang, L. Wang. and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106. 09685, 2021.
[26] G. Ilharco, M. Wortsman, R. Wightman, C. Gordon, N. Carlini, R. Taori, A. Dave, V. Shankar,
H. Namkoong, J. Miller, H. Hajishirzi, A. Farhadi, and L. Schmidt. Openclip, July 2021. URL If you use this software, please cite it as
https://ddi.org: below.
5281/zenodo 5143773
[27] Y. Jiang, V. Nagarajan, C. Back, and Z. Kolter. Assessing generalization of sgd via disagree- ment. International Conference Learning Representations, 2022.
[28] W. Koh, S. Sagawa, H. Marklund, M. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Ya- sunaga, R. L. Phillips. I. Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning, pages 5637-5664. PMLR, 2021.
[29] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Technical Report, 2009.
[30] I. Kuzborskij and F. Orabona. Stability and hypothesis transfer learning. In International Conference Machine Learning, pages 942-950. PMLR, 2013.
[31] D. Lee, B. Moniri, X. Huang, E. Dobriban, and H. Hassani. Demystifying disagreement-on-thee- line in high dimensions, 2023.
[32] Lewis, B. Oguz, R. Rinott, S. Riedel, and Schwenk. Miqa: Evaluating cross-lingual extractive question answering. arXiv preprint arXiv:1910.07475, 2019.
[33] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 1550-1999 2022.
[34] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint 17X 2017.
[35] O. Madani, D. Pennock, and G. Flake. Co-validation: Using model disagreement on unlabeled data to validate classification algorithms. Advances in neural information processing systems,
17, 2004.
[36] A. Malinin, N. Band, Ganshin, Alexander, G. Chesnokov, Y. Gal. M. J. F. Gales, A. Noskov, A. Ploskonosov, L. Prokhorenkova, Provilkov, V. Raina, V. Raina, Roginskiy, Denis, M. Shma- tova, P. Tigas, and B. Yangel. Shifts: A dataset of real distributional shift across multiple large-scale tasks, 2022.
[37] Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms. arXiv preprint arXiv:0902.3430, 2009.
[38] J. Miller, K. Krauth, B. Recht, and L. Schmidt. The effect of natural distribution shift on question answering models. In International conference on machine learning, pages 6905-6916. PMLR, 2020.
[39] J. P. Miller, R. Taori, A. Raghunathan, S. Sagawa, P. W. Koh, V. Shankar, P. Liang, Y. Carmon, and L. Schmidt. Accuracy on the line: on the strong correlation between dfitr--diittitution and in-distribution generalization. In International Conference on Machine Learning, pages 7721-7735. PMLR, 2021.
[40] P. Nakkiran and Y. Bansal. Distributional generalization: A new kind of generalization. arXiv preprint arXiv:2009, 200009 2020.
[41] A. Podkopaev and A. Ramdas. Distribution-Frr uncertainty quantification for classification under label shift. In Uncertainty in artificial intelligence, pages 844 453. PMLR, 2021.
[42] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
12

---

[43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages
PMLR, 2021.
[44] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint vrxiv:/606.052500 2016.
[45] B. Recht, R. Roelofs, L.. Schmidt, and V. Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprint arXiv: /80600041, 2018.
[46] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning, pages 5389-5400. PMLR, 2019.
[47] R. Roclofs, V. Shankar, B. Recht, S. Kridovich-Keil, M. Hardt, J. Miller, and L. Schmidt. A meta-analysis of overfitting in machine learning. Advances in Neural Information Processing Systems, 32, 2019.
[48] O. Russakovsky, J. Deng, H. Su, Krause, S. Satheesh, Ma, Z. Huang, A. Karpathy, A. Khosla, M. S. Bernstein, A. C. Berg, and L. Fei-Fei. Imagenet large scale visual recognition challenge. CoRR, abs/1409.057)
2014. (tt.:tt:::/rrrrt
org/ abs/ 1409. 0575
[49] S. Schelter, T. Rukat, and F. Biessmann. Learning validate the predictions of black box clas- sifiers on unseen data. In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. page 1289-1299, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450367356.
[50] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114. PMLR, 2019.
[51] R. Taori, A. Dave, V. Shankar, N. Carlini, B. Recht, and L. Schmidt. Measuring robustness to natural distribution shifts in image classification. Advances in Neural Information Processing Systems, 33:18583-18858 2020.
[52] R. Taori, Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto. Stanford alpaca: An instruction-followin Ilama model. https://githunn com Stanford_alpace 2023.
cww/tatsu. ttt//attttuu/uauaeeeee lab/
[53] H. Touvron, Martin, K. Stone, Albert, A. Almahairi, Y. Babaei, Bashlykov, Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. C. Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, Fu, W. Fu, B. Fuller, C. Gao, Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, I. Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, I. Molybog, Y. Nie, A. Poulton, J. Reizenstein, R. Rungta, K. Saladi, A. Schelten, R. Silva, E. M. Smith, R. Subramanian, X. E. Tan, B. Tang, R. Taylor, A. Williams, X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and Scialom. Llama Open foundation and fine-tuned chat models, 2023.
[54] D. Tran, J. Liu, M. W. Dusenberry, D. Phan, M. Collier, Ren, K. Han, Z. Wang, Z. Mariet, H. Hu, N. Band, T. G. J. Rudner, K. Singhal, Z. Nado, J. van Amersfoort, A. Kirsch, R. Jenatton, N. Thain, H. Yuan, K. Buchanan, K. Murphy, D. Sculley, Y. Gal, Z. Ghahramani, Snoek, and B. Lakshminarayanan. Plex: Towards reliability using pretrained large model extensions, 2022.
[55] T. H. Trinh and Q V. Lc. A simple method for commonsense reasoning. arXiv preprint WwXiv:1806002847 2018.
[56] H. Venkateswara, J. Eusebio, S. Chakraborty, and S. Panchanathan. Deep hashing network for unsupervised domain adaptation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5018-5027, 2017.
[57] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In ICLR, 2018
[58] D. Wang, X. Wang, L. Wang, M. Li, Q. Da, X. Liu, X. Gao, J. Shen, J. He, T. Shen, et al. Medfme: A real-world dataset and benchmark for foundation model adaptation in medical image classification. arXiv preprint arXiv:2306.0995 2023.
13

---

[59] A. Williams, N. Nangia, and S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference, 2018.
[60] M. Wortsman, Ilharco, W. Kim, M. Li, S. Korblithh, Roelofs, G. Lopes, H. Hajishirzi,
A. Farhadi, H. Namkoong, et al. Robust fine-tuning of zero-shot models, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7959-7971, 2022.
[61] C. Yadav and L. Bottou. Cold case: The lost mnist digits. Advances in neural information processing systems, 32, 2019.
[62] Y. Yu, S. Bates, Y. Ma, and M. Jordan. Robust calibration with multi-domain temperature scaling. Advances in Neural Information Processing Systems, 35:27510-27522. 2022.
[63] Y. Yu, Z. Yang, A. Wei, Y. Ma, and J. Steinhardt. Predicting out-of-distribution error with the projection norm, 2022.
[64] E. Zaken, S. Ravfogel, and Y. Goldberg, Bitfit: Simple parameter--effiiien fine-tuning for hunsformer-based masked language-models. arXiv preprint arXiv:2106. 10/99, 2021.
[65] S. Zhang. S. Roller, N. Goyal, M. Artetxe. M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin. T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, S. Koura, A. Sridhar, T. Wang, and L. Zettlemoyer. Opt: Open pre-trained transformer language models, 2022.
[66] Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27, 2015.
[67] Y. Zou, W. Deng, and I.. Zheng. Adaptive calibrator ensemble for model calibration under distribution shift, arXiv preprint arXiv:2303.0533/ 2023.
14

---

A
Appendix
Contents
A.1
Details Regarding OOD Performance Estimation Baselines
16
A.I.J
AGL-Based Estimation Methods; ALine-S/D
16
A.1.2
Temperature Scaling for Seffidenne-Basse Estimation Methods
17
A.1.3
Comparison to ProjNorm
17
A.1.4
Failure Datasets with Low Linear Correlation
18
A.2
Finetuning Hyperparameters
19
A.2.1
Linear Probing over CLIP for Vision Tasks
19
A,2.2
Full Finetuning GPT, OPT, BERT on Language Tasks
19
A.3
More Experiments on the Effect of Diversity Source with CLIP
21
A.3.1
OfficeHome Linear Probing Diversity Experiments
21
A.3.2
AGL appears in Random Head CLIP Ensembles across Datasets
22
A.3.3
Diversity Matters in Full Finetuned CLIP Ensembles)
26
A.3.4
Any diverse ensemble display AGL in models heavily trained from scratch
27
A.3.5
AGI. in FMs under Zero-Shot and Few-Shot Settings
27
A.3.6
Effect of Training Dataset Size
28
A.4
More Experiments on the Effect of Diversity Source for Extractive Question Answering
30
A.4.I
Single-Base Diversity Experiments Using OPT and BERT
30
A.4.2
Diversity in Full Finetuned GPT2 for Different Data Portions
A.333
Diversity in Full Finetuned OPT for Different Data Portions
34
A.4.4
Diversity in Full Finetuned BERT for Different Data Portions
38
A.5
More Experiments on the Effect of Diversity Source for Text Classification
42
A.5.I
Diversity Source for Linear Probing
42
A.6
Generative Question-Answering
43
A.6.1
Experiments the effect of Diversity Source
43
A.6.2
Experiments starting from multiple foundation models
45
Diversity under different learning rates and batch sizes
46
A.8
Diversity Experiments for different PEFT methods
47
15

---

A.1 Details Regarding OOD Performance Estimation Baselines
A.1.1 AGL-Based Estimation Methods: ALine-S/D
ALine algorithms are the AGL-based performance estimation methods proposed in [2]. When the AGL phenomenon occurs, i.c.. models observe a strong linear correlation in both ID versus OOD agreement and accuracy with matching slopes and biases, algorithms ALine-S and ALine-D effectively apply the linear transformation calculated using agreements to map the ID performances OOD performance estimates. We describe the algorithms more detail below.
AGL Provided a collection of models F = (2,,/ accuracy observe strong linear correlation if and only if ID versus OOD agreement observes strong linear correlation and when they do, the slopes and biases match: Vfs. fj E F where +
[n]. AGL suggests that ID versus OOD
+ (AccooD
+ (-((AccD((F)) +
1
(3)
- (Agroo = a. (Agr +
-- is the probit transform used to induce a better linear fit as used in [2] and [39]. Provided access Acci(ff), Ag fz), ofi fz) j. we'd like estimate ACCOOD for all fi F.
The algorithm ALine-S simply estimates the the slope a and bias of accuracy by computing the linear fit of agreement.
ALine-S
à, b arg min
E (Arr + (f. -b)
(4)
a.RER G
With à and 6, we estimate ACcooD(fi) 2 â Accip (ft) + b.. This method is called Aline-S.
ALine-D This method instead constructs the following system of linear equations. Provided the relation in Equation 2 derive that for any Si, E F,
1 ( "AAcooD (f+)) + - ( (Accooo(fa)))
1 Φ- +--((Acm(())) Φ -- (Agr fz) Φ (Agrooo + à ( (Accm((c +
Treating AccoOD (f)) Vi unknown variables, that the right hand side known and we can construct a linear system of equations using all (3) pairs of models. The algorithm employs least squares to solve this approximate system of linear equations.
16

---

A.1.2 Temperature Scaling for Confidence-Based Estimation Methods
We compare ALine against confidence-basee methods ATC 19, AC [24], and Doc-Feat 21 These methods notably perform better after calibrating the models ID by temperature scaling.
Classification For classification tasks, we optimize temperature T' for each model f on the cross-entropy loss over the in-distribution validation data.
Σ (E(((((()))p(())...)
min T
where (-) is the softmax.
Question-Answering For extractive question-answering tasks, the model has to predict two labels the start and end token index y [y', Me] of the context span that answers the question. For each model, attach span prediction head [Ua: Vel Rdx2 top of the base BB(x) E RR where N is the token length of æ. s(z) vs Bx(x) and e(x) te BB(x) predict the start and end token index, respectively.
We' re interested in evaluating question question-answeene answering models on the exact match (EM) objective, EM y) = [ū- yul
lāe yel
EM treats question question-answeriin answering classification problem N x N choices of start and end index pairs. This allows us utilize confidence-bil methods that are designed for classification tasks. We calculate the model confidence for index pair [i,j] (s(());; (((()))
We jointly optimize separate temperature for the start and end logits T, and Te and we minimize the cross-entropy loss over the in-distribution validation data.
min Σ
(8)
A.1.3 Comparison to ProjNorm
In this section, we present a comparison with ProjNorm 663 a method that yields a score which is shown to be correlated with the OOD performance of the model. We study the same setting as presented in Section 4 where we estimate OOD performance of foundation models pretrained on different text corpora. Unlike AGL, ProjNorm doesn't 't provide estimate of the OOD performance hence we compare the linear correlation between the predicted value and OOD performance. From Ta- ble 4 it can be seen that estimates from ALine-D - are more strongly correlated with OOD performance than Porj.
Table 4: Correlation coefficient between OOD Accuracy and Prediction for ALine-D and Proj SQuAD-Shifts Amazon SQuAD-Shifts Reddit
Method
ALine-D
0.98
0.98
ProjNorm 0.64
0.79
17

---

A.1.4 Failure Datasets with Low Linear Correlation
In our comparison with baselines in Section 5 we filter out datasets with low correlation coefficient < 0.95 ID OOD agreement. When the linear correlation agreement is weak, AGL tells us that the correlation is also low for accuracy, and AGL-based methods are not guaranteed to be reliable such circumstances. We provide ID OOD accuracy and agreement scatter plots for all datasets in Appendix A.3.2
Below, separately provide the comparison with baselines for the excluded datasets. We generally find that the baseline ATC 19 is significantly better in circumstances where AGL-based methods unreliable.
Table 5: The MAPE (%) of predicting OOD performance using AGL-based ALine and other baseline methods. We collect diverse ensemble by randomizing the linear initialization and including multiple base models.
OOD Dataset
Agreement R2
ALine-D
ALine-S
Naive Agr
ATC
AC
DOC-Feat
CIFARIOS 10C Gaussian Noise
0.85
65.59
56.67
41.85
42.77
74.60
75.40
CIFARIOS IOC Glass Blur
0.89
44.24
44.97
31.24
33.58
79.45
80.36
JFC Shot Noise
0.89
45.02
37.60
27.70
28.25
49.23
49.90
CIFARIOC Speckle Noise
0.90
36.67
30.13
23.45
22.72
43.46
44.12
CIFAR 1OOC Gaussian Noise
0.93
34.98
32.21
28.88
21.18
69.04
63.81
CIFAR 100C Glass Blur
0.93
66.03
63.45
42.41
25.47
109.73
103.70
ImageNerC Gaussian Noise
0.89
50.10
47.26
73.47
13.21
54.86
42,66
ImageNetC Glass Blur
0.87
74.20
71.84
100.32
15.97
74.12
59,60
Iet, Impulse Noise
0.88
62.86
59.68
87.83
16.54
63.23
49.99
ImageNerC: Shot Noise
0.88
53.89
51.12
78.37
14,66
57.47
44.58
iWildCam- CLC----WWLDSSS WILDS
0.85
22.05
25.29
46.42
37.25
57.31
69.58
Camelyon1 7-WLDDS
0.59
10.14
6.44
13.26
6.46
8.76
8.90
18

---

A.2
Finetuning Hyperparameters
We state here the hyperparameters used finetune the models for diversity experiments reported in Section 3
A.2.1 Linear Probing over CLIP for Vision Tasks
We train all linear probes using SGD. Models trained for different timesteps achieve an distribution of ID accuracies.
Table 6: CLIP Linear Probing
Dataset
Hyperparameters
Learning Rate: 5 10
CIFAR10
Batch Size: 1028
Learning Rate: 1 10-1
CIFARI00
Batch Size: 1028
Learning Rate: 1 10
ImageNet
Batch Size: 1028
Learning Rate: 1 10-3
OfficeHome
Batch Size: 200
FMoW-
Learning Rate: 1 x 10
WILDS
Batch Size: 200
iWildCam-
Learning Rate: 1 x 10
WILDS
Batch Size: 200
Camelyon17-
Learning Rate: 10-3
WILDS
Batch Size: 200
A.2.2 Full Finetuning GPT, OPT, BERT on Language Tasks
We AdamW [34] full finetune language models. We keep the learning rate small. Models trained for different timesteps to achieve an even distribution of ID accuracies.
Table 7: Full Finetuning GPT2-Medium on SQuAD
Source of Diversity
Hyperparameters
Learning rate: x 10
Weight Decay: 1 x 10-5
Initialization Ordering
Batch Size: 32
Max Epochs: 20
Learning rate: 6, 4 x 10-7
Weight Decay: 1 x 10-5
Subsetting (10, 30 % of data)
Batch Size:
Max Epochs: 20
19

---

Table Full Finetuning OPT-125M on SQuAD
Source Diversity
Hyperparameters
Learning rate: 4 x 10-7
Weight Decay: 10-
Initialization + Ordering
Batch Size: 32
Max Epochs: 20
Learning rate: 40, 12, 8 x 10
Weight Decay: 10
Subsetting (10, 30, 50 % of data)
Batch Size: 32
Max Epochs: 20
Table 9: Full Finetuning BERT-Uncased on SQuAD
Source of Diversity
Hyperparameters
Learning rate: 2 x 10-?
Weight Decay: 10
Initialization + Ordering
Batch Size: 32
Max Epochs: 20
Learning rate: 20, 6, 4 10-7
Weight Decay:
10 10-5
Subsetting (10, 30, 50 % of data)
Batch Size: 32
Max Epochs: 20
Table 10: Full Finetuning GPT2-Medium on MNLI
Source of Diversity
Hyperparameters
Learning rate: 5 10-4
Weight Decay: x 10-5
Initialization + Ordering
Batch Size: 128
Max Epochs: 10
Learning rate: 5 X 10
Weight Decay: x 10
Subsetting (10% of data)
Batch Size: 128
Max Epochs: 10
Table 11: Full Finetuning OPT-125M on MNLI
Source of Diversity
OPT-125M
Varied
Fixed
Learning rate: x 10-3
Weight Decay: 1 x 10-5
Initialization Ordering
Batch Size: 128
Max Epochs: 10
Learning rate: 1 x 10-2
Weight Decay: 1 x 10
Subsetting (10% of data)
Batch Size: 128
Max Epochs: 10
20

---

A.3
More Experiments on the Effect of Diversity Source with CLIP
A.3.1
OfficeHome Linear Probing Diversity Experiments
In Section 3.1 we examine how diversity source impacts whether AGL is observed in linear probed CLIP models from CIFARIO to CIFARIOC. Here, we perform the same experiment on Office-Home [61. which consists of domains or image styles ("Art", "Clip Art", "Product", and "Real World") for 65 common objects. We train models on one domain and treat the remaining three domains OOD, Similarly, only Random Initialization yields AGL matching slopes accuracy and agreement, and as a result, the corresponding MAPE of estimating the OOD performance of this diverse ensemble the smallest.
Officettome Clipart
Officettome Clipart
OfficeHome Clipart
Accurscy Agtostment
Accuracy
AccltaCy Agrezment
000
000
10
10
OfficeHome Real
OfficeHome Real
OfficeHome Real
Arcunecy Agreement
000
000
52
Officellome Art
OfficeHoome
OfficeHlome Art
Accuncc Agreement ent:
Accuray Agresment
OOD
32
50
70
10
70
90
53
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 4: ID vs OOD accuracy and agreement of linear probed CLIP models on OfficeHome Art (top row), Product (middle row), and Real World (bottom row). The figure title is the OOD domain.
Table 12: The S MAPE(%) for ensembles trained on each domain of OfficeHome.
OfficeHome ID domain
Source of Diversity
OOD Estimation MAPE(S)
Random
Data Ordering
20.67
Subsetting
120.85
ClipArt
Random Linear Heads
12.23
Data Ordering
28.45
Data Subsetting
78.49
Product
Random Linear Heads
13.63
Data Ordering
110.45
Data Subsetting
92.97
Real
Random
Data Ordering
33.36
Data Subsetting
3078
21

---

A.3.2 AGL appears in Random Head CLIP Ensembles across Datasets
We report the strength of AGL in linear probed CLIP models with randomly initialized linear heads across all datasets discussed in Section 3.2
CIFARIOC (Brightness)
CIFAR1OC (Contrast)
CIFAR1OC (Defocus Blur)
CIFRROOC (E (Elastic Transfom))
Accumaoy Agreement
Actuacy
Accurayy Agreement
Accuray Agreement
Agreemant
10
10
10
CIFARIOC (Fog)
CIFAR1OC (Frost)
CIFARIOC (Gaussian Blur)
CIFARIOC (Gaussian Noise)
Arcumey
Actuaac Agreemat:
Acurayy Agisemant
Agcurey
Age cemmont
Agrcemist
000
000
1D
10
CIFRROCC (Glass Blur)
CIFAR10C (Impuise Noise)
CIFAR1OC (JEG Compression)
CIFAR10C (Motion Blur)
Accurocy
Acor jCy
Acorrry Agreement
Accuracy
Age Agrsement
Agreemant
Agroement
000
000
10
10
CIFARIOC (Pixel ate)
CIFAR1OC (Saturate)
CIFAR10C (Shot Nolse)
CIFARIOC (Show)
Accureey Aggeement
Accu Acmeement
Accaracy Agreemere
Accurecy
meetment
000
50
000
13
1D
1D
CIFARIOC (Spatterr)
CIFAR1OC (Speckle Noise)
CIFARIOC (Zoom Blur)
Accuracy
Accuracy
Accurcc
Agreement
Agreement
Agroement
73
000
30
000
000
10
Figure 5: AGL and ACL for all CLOC shifts with random head initialization finetuning.
22

---

CIFARI0.1
Accarayy Agreenent
Acarway Agrrement
OCD
10
10
Figure 6: AGL and ACL for the C10.1 shifts with random head initialization finetuning.
CIFAR1ODC (Brightness)
CIFAR10OC (Contrast)
CIFAR10OC (Defacus Blur)
CIFAR (Elastic Transform)
Actuacy Agrccmatt
Acournyy Agrccment
Aaturayy commmtt
Agrnement
70
000
19
23
15
16
1D
10
CIFAR10OC (Fag)
CIFARIGOC (Frost)
CIFRR10OC (Gaussian Blur)
CIFAR1ODC: (Gaussian Naise)
Ascurady
Acoracy
Actuacyy Agruement
AcCuracy
Agrseman:
30
000
30
900
20
23
35
56
10
10
10
10
CIFAR10OC
CIFAR10OC (impuise
CIFAR100C Compression)
CIFAR10OC (Motion
Acuraayy
Accrrsy Apreeman
Asuraay Agreament
Accurony Agroement
Agresment
000
53
05
35
76
06
10
10
10
CIFAR10OC (Pixelate)
CIFAR100C (Saturate)
CIFAR10OC Noise)
Aecumey
Actuacyy
Annracy Agreement
Acturacy
Aggaement
Aptrerenn
fggnement
30
000
000
35
16
10
10
10
CIFAR100C (Spatber)
CIFAR100C (Speckle Noise)
CIFAR100C (Zoom Blur)
Accuracy
AcAM 311
Accurocy
Agresment
Agreammntt
Agrosment
000
000
000
30
88
56
33
33
06
10
10
Figure AGL and ACL for the C100C shifts with random head initialization finetuning.
23

---

ImageNetC (Brighlness)
ImageNetC (Contrast)
ImageNelC
Accumcy
Accuracy cementt
Accuraity Agresment
Agresment
8
10
10
ImageNetC (Fog)
ImageNetC (Frost)
ImageNett (PEG Compression)
20
00D
60
Figure 8: AGL and ACL for the ImageNetC shifts with random head initialization finetuning.
ImageNelv2 Top Images
ImageNetv2 Threshold 0.7
ImageNelv2 Matched Frequency
Acouniy
Accuracy cemnent
Accurazy Agrezment
Agresment
30
Figure 9: AGL and ACL for the ImageNet V2 shifts with random head initialization finetuning.
Camelyon17-WiLDD (OOD)
1MoW-WILDS (OOD)
WWlldCam- DILDD (OOD)
Accumcy
Agresmment
Agresment
30-
Arcutwty Agnenment
30
10
36
56
10
56
10
Figure 10: AGL and ACL for benchmarks from the WILDS dataset with random head initialization finetuning.
Officellome Clipart
Officelamme Product
OfficeHome Real
Agresment
10-
900
000
000
32
Figure 11: AGL and ACL for the OfficeHome ClipArt, Product, Real shifts with random head initialization finetuning over OfficeHomm Art.

---

Hooe
Accuracy Agrezmentt
surrent
70
cstrent
0OD
00D
000
ID
10
1D
Figure 12: ID vS OOD accuracy and agreement with OH ClipArt ID
Figure 13: AGL ACL for the OfficeHome Art, Product, Real shifts with random head initialization finetuning OfficeHome ClipArt.
OfficeHtome Art
Officettome Clipart
OfficeHome Real
16
00D
000
100
18
Figure 14: AGL and ACL for the OfficeHome ClipArt, Art, Real shifts with random head initialization finetuning over OfficeHome Product.
OfficeHome
OfficeHome Clipart
Officettome Product
Accuracy Agresminnt
sommennt
Agrenment
00D
50
AdP
000
50
000
5/
Figure 15: AGL and ACL for the OfficeHome Art, ClipArt, Product shifts with random head initialization finetuning over OfficeHome Real.
25

---

A.3.3 Diversity Matters in Full Finetuned CLIP Ensembles
We show that light full finetuning over CLIP also observes similar effects of diversity source on the strength of AGL. We verify this on shifts from CIFARIO CIFARIOC.
CIFARIOC (JEG Compression)
CIFARL0C Compression)
CIFARIOC Compression)
Accuracy Agi sumrent
50
20
OGD
000
10-
50
-0
32
10
ID
10
10
CIFAR1OC (Pixelate)
CIFARIOC (Pixelate)
CIFARI0C (Pixelate)
Arruarr
Accuntry
Sessmmntt
Agreement
Agremmeent
000
000
13
33
ID
1D
(a) Random Head
(b) Ordering
(c) Data Subsetting
Figure 16: ID vs OOD accuracy and agreement of full finetuned CLIP models on shift from CIFARIO to CIFARIOC "JPEG Compression" (top row) and "Pixelate" (bottom row) shifts. Only Random Initialization (Column a) yields AGL matching slopes in accuracy and agreement.
Table The average MAPE(%) ALine estimates of OOD performance of full finetuned CLIP models across all 19 CIFAR10C shifts. As can be seen from Figure 4 only ensembles with diverse random initialization consistently results in smallest MAPE values.
Source of Diversity
CIFARIOC MAPE(%)
Random Linear Heads
29.37
Data Ordering
69.98
Data Subsetting
33.57
26

---

A.3.4 Any diverse ensemble display AGL in models heavily trained from scratch
On the other hand, we demonstrate that in models heavily trained from scratch, AGL can be observed irrespective of the diversity source. Consistent with the models in [2], we train ResNet18 on CIFAR10 from scratch, varying the different sources of randomness. These models are trained heavily with SGD with learning rate of 10 batch size 128, and weight decay of 10 for up 200 epochs. We do not use any data augmentation.
Init
Order
Subset
0.7
0.7
0.5
00D
COD
0.5
0.5
0.3
0.3
0.7
ID
ID
ID
Figure 17: Effect of Diversity Source on ResNet18 from CIFARIO to CIFAR10C-Snon
A.3.5
AGL in FMs under Zero-Shot and Few-Shot Settings
Zero-Shot OLMo7B
Few-Shot LP over CLIP
Agr R20.892
Agr R20. 988
5QuADshifts Reddit
Acc R20.909
CIFAR10C Pixelate
Acc R20.994
70
30-
30
10
30
70
90
50 70 CIFAR10
90
SQuAD
Figure 18: Zero-Shot and Few-Shot Settings. (Left) We plot the ID versus OOD zero-shot per- formances of OLMo7B checkpoints. We see that the linear correlation is weak both and F1 F1-Agreement. This means that the effective robustness of base models vary widely during pretrain- ing. (Right) We train linear probes over CLIP embeddings on few-shot CIFARIO (10 examples per class) with random initialization. We similarly observe AGL and ACL with random initialization.
27

---

A.3.6 Effect of Training Dataset Size
To rid of any confounding factors from data subsetting observing a smaller amount of data, we evaluate all randomness sources on different training dataset sizes. We track the effect of diversity in random initialization, data ordering, and data subsetting for different portions of the training data (100%. 50%, 30%, 10%). For each percentage x%, Random Initialization and Data Ordering ensembles are trained on the same randomly sampled 2% proportion of the data while each model in the Data Subsetting ensemble observe different randomly sampled 2% portions
CIFAR1OC (JEG Compression)
CIFARIOC Compression)
Accursty
Ag remmotk
75-
000
10
1D
1D
CIFAR10C (Pixelate)
CIARIOC (Pixelate)
Accurscy
Ag rereement
30
40 60
ID
ID
(a) Random Head
(b) Data ordering
Figure 19: ID OOD accuracy and agreement of linear probe CLIP models finetuned 100% of CIFARIO training data evaluated on the CIFARIOC "JPEG Compression" (top row) and "Pixelate" (bottom row) shifts
CIFARIOC (JPEG Compression)
CIFAR10C (JPEG Compression)
CIFARIOC (JPEG Compression)
Accuraiy
Accuracy
Agtesment
eemmetttt
Agreement
000
000
00
5.-
ID
CIFAR1OC (Pixelate)
CIFAR10C (Pixelate)
CIFAR10G DC (Pixelate)
26
Accuracy
Accuracy
apreemeent
asmrent
Agrocmentt
DOD
000
(a) Random Head
(b) Data ordering
(c) Data Subsetting
Figure 20: ID vs OOD accuracy and agreement of linear probe CLIP models finetuned on 50% of CIFAR10 training data evaluated on the CIFAR1OC "JPEG Compression" (top) and "Pixelate" (bottom) shifts
28

---

CIFARIOC (JPEG Compression)
CIFARIOC (JPGG Compression)
CIFARIOC (JPEG Compression)
Accurciy
Accuraty Agresment
agremment
eetment
000
000
ID
CIFARIOC (Pixelate)
CIFARIOC (Pixelate)
CIFAR10C DC (Pixelate)
26
Accuracy
Accuracy Agrozmintt
Agrecmment
DOD
000
DOD
(a) Random Head
(b) Data ordering
(c) Data Subsetting
Figure 21: ID vs OOD accuracy and agreement of linear probe CLIP models finetuned on 30% of the CIFAR10 10 training data evaluated on the CIFARIOC "JPEG Compression" (top) and "Pixelate" (bottom) shifts
CIFARIOC (JPEG Compression)
CIFAR10C (JPEG Compression)
CIFARIOC (JPEG Compression)
Arcurety
Accurayy
apresment
Agresment
000
000
000
50
10
ID
CIFAR1OC (Pixelate)
CIFAR10C (Pixelate)
CIFAR10C (Pixelate)
16
Accuracy comnent
Accuracy
Agresmrent
00D
10
300
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 22: ID vs OOD accuracy and agreement linear probe CLIP models finetuned on 10% the CIFAR10 training data and evaluated on the CIFARIOC "JPEG Compression" (top) and "Pixelate" (bottom) shifts
29

---

A.4 More Experiments on the Effect of Diversity Source for Extractive Question Answering
A.4.1 Single-Base Diversity Experiments Using OPT and BERT
In Section 3.2 report the effect of diversity source in full finetuned GPT2-Medium models for the extractive QA task SQuAD to SQuAD-Shifts. In this section, we also provide the same experiments on OPT-125M and BERT. As observed in GPT, using Random Heads yields the strongest AGL behavior and achieves the smallest ALine-D MAPE.
report the MAPE of OOD performance estimation of models trained with Random Intiialization and Data Ordering using 100% of training data and Data Subsetting with 10% of training data.
In Table 14 and 15
Table 14: The average MAPE (%) of ALine-D OOD performance estimates of OPT-125M models full finetuned on SQuAD.
Source of Diversity SQuAD-Shife Amazon
SQuAD-Shifte Reddit
Random Head
6,54
5.43
Data Ordering
11.37
8.70
Data Subsetting
11.15
9.65
Table 15: The average MAPE (%) of ALine-D OOD performance estimates of BERT models full finetuned on SQuAD.
Source of Diversity SQuAD-Shifts Amazon
SQuAD--hifts Reddit
Random Head
14.70
8.62
Data Ordering
9.16 16.55
Data Subsetting
22.13
18.64
30

---

A.4.2 Diversity in Full Finetuned GPT2 for Different Data Portions
We track the effect of diversity in random initialization, data ordering, and data subsetting for different portions of the training data (100%, 50%, 30%, 10%). For each percentage x%, Random Initialization and Data Ordering ensembles are trained on the same randomly sampled x% proportion of the data while each model in the Data Subsetting ensemble observe different randomly sampled 2% portions.
SQuAD-Shifts (Am
SQUAD-Shifts (Amazon)
F1 Score
F1 Soae
Ag oeTHLrk
Ajtvemart
70-
72-
30
50
70
30
10
1D
SQuAD-Shifts (Reddit)
SQuAD-Shitts (Reddit)
Score
Scare
Ag nemmmon
Agreament
50
52-
SQUAD-Shi Shifts (New York Times)
SQuAD-Shits (New York Times)
F1 Score
Agreement
32
10
10
SQuAD- Shifts (New Wiki)
SQuAD Shifts (New Wiki
+1 Score
F1 Scor Ajtvemont
Ag nemunt
70
72
30
1.0
75
10
70
(a) Random Head
(b) Data Ordering
Figure 23: ID vs OOD accuracy and agreement of models finetuned on SQuAD from a single pretrained GPT2 model with 100% of the training data
31

---

SQuAD-Shifts (Amazon)
SQuAD-Shifs (Amazon)
SQuAD-Shifts (Amazon)
Score
20
Scone
Sccmm Agresment
Ag aomesment
Agresment
000
000
08
50
35
50
30
SQuAD-Shifts (Reddit)
SQuAD-Shifts (Reddit)
SQuAD-Shifts (Reddit)
FI Score
FI Scor
Agrosr resmmeett
Agnosment
16-
72
000
50
53
10
10
10
SQuAD-Shifts (New Yark Times)
SQuAD-Shifts (New Yark Times)
SQuAD-Shii (New York Times]
Score Agreemment
Agreement
00D
600
000
1D
SQuAD-Shifts
SQuAD Shifts (New Wiki)
SQUAD-Sun Shif-S (New Wiki)
SQuAD-Shin D-hhitt--tts (New Wiki)
50
50
Score
oumeent
Agreemeen
DOD
000
30
35
56
30
32
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 24: ID vs OOD accuracy and agreement of models finetuned on SQuAD from single pretrained GPT2 model with 50% of the training data
32

---

SQuAD-Shifts (Amazon)
SQuAD-Shitt (Amazon)
SQuAD-Shifts (Amazon)
Score
Scone
Sccmm Agresment
Ag mesmmeett
Agresment
0000
000
50
33
30
32
SQuAD-Shilts (Reddit)
SQuAD-Shifts (Reddit)
SQuAD-Shifts (Reddit)
FI Score
FI Scor
Agresr resmmment
Agnesment
16-
72
00D
000
000
50
53
10
10
SQuAD-Shifts (New Yark Times)
SQuAD-Shiff Shiifs (New Yark Times)
SQuAD-Shife (New York Times)
Score
Fl Stom Agrosmantt
retemment
Agr agrnemment
0OD
100
600
54
1D
SQuAD Shifts SQuAD-Shifts
(New Wiki)
SQuAD Shifts (New Wiki)
CQuAD-Shifts
SQuAD-Shifts Shifts (New Wiki)
56
F1 Score
50
comennt
DOD
0000
30
23
30
30
32
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 25: ID vs OOD accuracy and agreement of models finetuned on SQuAD from single pretrained GPT2 model with 10% of the training data
33

---

A.4.3 Diversity in Full Finetuned OPT for Different Data Portions
We track the effect of diversity in random initialization, data ordering, and data subsetting for different portions of the training data (100%, 50%, 30%, 10%). For each percentage x%, Random Initialization and Data Ordering ensembles are trained on the same randomly sampled x% proportion of the data while each model in the Data Subsetting ensemble observe different randomly sampled 2% portions.
SQuAD-Shifts (Am azon)
SQuAD-Shift (Amazon)
F1 Scors
Ag oeTHLrk
70-
30
70
30
32
10
ID
SQuAD-Shifts (Reddit)
SQuAD-Shits (Reddit)
13 Score
Ag nemmmot
Agreament
50
10
SQUAD-SMI Shifts (New York Times)
SQuAD-Shitt (New York Times)
F1 Score
Agreement
32
10
10
SQuAD- Shifts (New Wiki)
SQuAD Shifts (New Wikil
+1 Score
Ag nemunt
70
000
33
1.0
75
10
22
1D
(a) Random Head
(b) Data Ordering
Figure 26: ID vs OOD accuracy and agreement of models finetuned on SQuAD from a single
pretrained OPT- 125m with 100% of the training data
34

---

SQuAD-Shifts (Amazon)
SQuAD-Shifs (Amazon)
SQuAD-Shifts (Amazon)
Score
Scone Agresment
Scom
Ag mesmmeett
Agresment
0000
000
50-
50
5.0
SQuAD-Shifts (Reddit)
SQuAD-Shilts (Reddit)
SQuAD- (Reddit)
FI Score
Agross resmment
Agreement
16
00D
50
12
50
10
10
10
SQuAD-Shifts (New Yark Times)
CQuAD-Shifts (New Yark Times)
SQuAD-Shitt Shifts (New York Times]
F2 Score
Score
tetemment
Agreemmen
Agreementt
00D
000
00D
28-
SQuAD Shifts SQuAD-Shifts
(New Wiki)
SQUAD-Shit Shiffs (New Wiki)
SQuAD Shifts (New Wiki)
SQuAD-Shifts
56
50-
F1 Pcore
csment
Agreemmnnt
Score
DOD
000
30
13
50
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 27: ID vs OOD accuracy and agreement of models finetuned on SQuAD from single pretrained OPT- with 50% of the training data
35

---

SQuAD-Shifts (Amazon)
SQuAD-Shits (Amazon)
SQuAD-Shifts (Amazon)
Score
Scone Agresment
Scom
mesmmnnt
Agrenment
0000
000
50
52
30
32
SQuAD-Shifts (Reddit)
SQuAD-Shilts (Reddit)
SQuAD SQuAD-Shifts (Reddit)
FI Score
Ag resmment
Agreement
16-
00D
000
08
54-
50
12
10
10
10
SQuAD-Shifts (New Yark Times)
SQuAD-Shifts (New Yark Times)
SQUAD Shitf (New York Times]
Score Agr agreemment
Score
aeremment
Agteementt
70
00D
00D
00D
20
1D
SQuAD Shifts SQuAD-Shifts
(New Wiki)
SQuAD-Shine Shif-s (New Wiki)
SQuAD Shifts (New Wiki)
SQuAD-Shifts
50
F1 Score
Score
csmennt
Agrocmmnnt
DOD
000
0000
30
13
30
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 28: ID vs OOD accuracy and agreement of models finetuned on SQuAD from single pretrained OPT-128 with 30% of the training data
36

---

SQuAD-Shifts (Amazon)
SQuAD-Shits (Amazon)
SQuAD-Shifts (Amazon)
Score
Scone Agresment
Sccmm
mesmmentt
Agresment
0000
000
08
50
30
30
SQuAD-Shifts (Reddit)
SQuAD-Shilts (Reddit)
SQuAD-Shifts (Reddit)
EI Score
FI Scorm
Agresrre resmment
Agnesment
16-
72
00D
00
50
12
30
53
10
10
1D
SQuAD-Shifts (New Yark Times)
SQuAD-Shifts (New Yark Times)
SQuAD-Shif (New York Times]
Score Agreemeen
aetemment
Agrwement
00D
00D
000
1D
SQuAD-Shifts
SQuAD Shifts (New Wiki)
SQUAD-Shit Shiff-s (New Wiki)
SQuAD-Shifts Shhihit (New Wiki)
56
F1 Score
50
cumennt
Agreemient
DOD
000
30
30
30
32
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 29: ID vs OOD accuracy and agreement of models finetuned on SQuAD from single pretrained OPT- with 10% of the training data
37

---

A.4.4 Diversity in Full Finetuned BERT for Different Data Portions
We track the effect of diversity in random initialization, data ordering, and data subsetting for different portions of the training data (100%, 50%, 30%, 10%). For each percentage x%, Random Initialization and Data Ordering ensembles are trained on the same randomly sampled x% proportion of the data while each model in the Data Subsetting ensemble observe different randomly sampled 2% portions.
SQuAD-Shifts (Am azonz
SQuAD-Shifts (Amazon)
F1 Scors
Ag oemHurk
70-
72
10
50
70
30
75
10
10
SQuAD-Shifts (Reddit)
SQuAD-Shifs (Reddit)
13 Score
Scar Agreament
Ag nemmmon
72-
50
30-
10
SQUAD-SMI Shifts (New York Times)
SQuAD-Shitt (New York Times]
50-
F1 Score
Agreement
32-
10
SQuAD- Shifts (New Wiki)
SQuAD Shifts (New Wikil
+1 Score
Ag nemunt
70
000
30
1.0
76
10
22
55
1D
(a) Random Head
(b) Data Ordering
Figure 30: ID vs OOD accuracy and agreement of models finetuned on SQuAD from a single pretrained BERT with 100% of the training data
38

---

SQuAD-Shifts (Amazon)
SQuAD-Shits (Amazon)
SQuAD-Shifts (Amazon)
Score
Scone Agresment
Sccmm Agresment
mesmmnnt
0000
000
50
30
55
70
30
SQuAD-Shilts (Reddit)
SQuAD-Shilts (Reddit)
SQuAD-Shifts (Reddit)
EI Score
FI. Scoccm
Agross resmment
Agnesment
16-
72
00D
50
54-
50
55
53
10
10
1D
SQuAD-Shifts (New Yark Times)
SQuAD-Shift Shhifs (New Yark Times)
SQuAD-Shif (New York Times]
F1 Score
aeremment
Agreemment
Agrwement
00D
000
000
1D
1D
SQuAD Shifts SQuAD-Shifts
(New Wiki)
SQUAD-Shit Shiff-s (New Wiki)
SQuAD-Shifts S-hihtts (New Wiki)
56
F1 Score
50
oumentt
Agreemient
DOD
000
30
30
53
30
32
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 31: ID vs OOD accuracy and agreement of models finetuned on SQuAD from single pretrained BERT with 50% of the training data
39

---

SQuAD-Shifts (Amazon)
SQuAD-Shifs (Amazon)
SQuAD-Shifts (Amazon)
Score
Scone Agresment
Sccmm
mesmment
Agresment
0000
000
50
50
50
30
SQuAD-Shifts (Reddit)
SQuAD-Shifts (Reddit)
SQuAD-Shifts (Reddit)
FI Score
FI. St.ccm
Agross resmment
eemeettntt
Agnesment
16-
72
00D
54-
58
50
12
30
53
10
10
1D
SQuAD-Shifts (New Yark Times)
SQuAD-Shiff Shiifts (New Yark Times)
SQuAD-Shiff (New York Times]
Score
retemment
Agrnemmen
Agrwement
00D
000
000
1D
SQuAD-Shifts
SQuAD Shifts (New Wiki)
SQUAD-Shin Shiff- (New Wiki)
SQuAD-Shifts D-hii-ts (New Wiki)
50
Fi Score
50
comenn
Agreemient
DOD
000
30-
30
13
30
32
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 32: ID vs OOD accuracy and agreement of models finetuned on SQuAD from single pretrained BERT with 30% of the training data
40

---

SQuAD-Shifts (Amazon)
SQuAD-Shifs (Amazon)
SQuAD-Shifts (Amazon)
Score
Scone Agresment
Sccmm Agresment
mesmment
0000
000
50
50
13
30
50
30
SQuAD-Shifts (Reddit)
SQuAD-Shilts (Reddit)
SQuAD-Shifts (Reddit)
EI Score
FI Scom
Ag resmment
Agnosment
16-
72
00D
00
54-
50
12
10
10
1D
SQuAD-Shifts (New Yark Times)
SQuAD-Shiff Shiffs (New Yark Times)
SQuAD-Shift (New York Times]
Score
Agreemment
Agreement
00D
600
000
1D
1D
SQuAD Shifts SQuAD-Shifts (New Wiki)
SQuAD-Shine Shift-s (New Wiki)
SQuAD-Shifts D-hii-ts (New Wiki)
56
F1 Score
50
oumennt
Agreemeent
DOD
200
30
30
50
30
32
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 33: ID vs OOD accuracy and agreement of models finetuned on SQuAD from single pretrained BERT with 10% of the training data
41

---

A.5
More Experiments on the Effect of Diversity Source for Text Classification
A.5.1 Diversity Source for Linear Probing
reported the effect of diversity source for full finetuned OPT. Here, demonstrate similar results on text classification shift from MNLI-matched to SNLI with linear probed GPT2- Medium (Figure 34) and OPT-125M (Figure 35 Similarly, random initialization observes the strongest AGL behavior. In Table T6. we report the average MAPE of OOD performance estimation using ALine across models.
In Section 3.2
SNLI
SNLI
SNLI
Accuracy
Accuracy
Agroumeent
Age cumrett
76
70
00D
00D
000
15
1.5
20
40
ID
10
10
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 34: ID vs OOD accuracy and agreement models finetuned on MNLI from GPT2-Medium. Random Head and Data Ordering ensembles trained on 100% of the training data while Data Subsetting ensemble is on 10%.
SNLI
SNU
SNLI
Accuracy
Accuracy
Accuracy
Agresmment
serment
Agresmentt
0.00
0000
000
16
10
30
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 35: ID vs OOD accuracy and agreement models finetuned on MNLI from OPT-125M. Random Head and Data Ordering ensembles are trained on 100% of the training data while Data Subsetting ensemble is on 10%.
Table 16: The average MAPE (%) of ALine-D performance estimates of accuracy SNLI.
Model
GPT2-Medium
OPT-125M
Random Linear Heads
5.9
5.4
Data Ordering
6.7
12.4
Data Subsetting
5.3
8.6
42

---

A.6
Generative Question-Answering
We also test whether AGL appears for generative question answering, where models generate the answer to the question instead of directly extracting span from the provided context.
A.6.1 Experiments on the effect of Diversity Source
For generative tasks, common to finetune the base model directly instead of attaching linear head on top. We again study different ways of achieving diverse set of classifiers (e.g.. Random Initialization, Data Ordering, Data Subsetting). However, we replace randomly initialization of a linear head instead with randomly initializing the non-zero initialized LoRA weights.
Experimental Details We base model GPT2 and finetune models using cross-entropy loss on the next token prediction objective over SQuAD. During training, we concatenate the context, question, and answer together, and apply the next token prediction objective just the answer tokens. We add a line break (In) after the answer, which functions as a "stop token" that the models also have to predict after answering the question. We train with the AdamW optimizer with learning rate of e 4, weight decay of le 2, and batch size of 16 up to a maximum of 4 epochs. As with extractive QA, we measure the F1 score of the model's answer (its output before line break).
Conclusion We track the effect of diversity for 100% the training data random initialization and data ordering, 50% of the training data in data subsetting. Figure [3 shows that all sources of diversity are sufficient to show AGL. Diversity is different from extractive drestion-answeringg because the search space of all tokens already provides enough diversity among models. We also note the MAE in Table I7 and MAPE in Table
Table 17: ALine-S MAE (%) of different sources of diversity for generative question-answering GPT2
Source of Diversity
SQuAD-Shifts Amazon
SQuAD-Shifts Reddit
SQuAD-Shifts New-Wiki
SQuAD-Shifts NYT
Random Lincar Heads
0.0354
0.0170
0.0129
0.0148
Data Ordering
0.0296
0.0177
0.0165
0.0152
Data Subsetting
0.0312
0.0205
0.0178
010166
Table 18: ALine-S MAPE (%) of different sources of diversity for generative question-answeriin on GPT2
Source of Diversity
SQuADD-hhits Amazon
SQuAD>-Shits Reddit
SQuAD--hiits New- Wik-Wiki
SQuAD-Shifts NYT
Random Lincar Heads
13.93
6.6
4.88
5.06
Data Ordering
12.05
5.91
5.41
Data Subsetting
16.43
10.08
7.47
8.32
43

---

SQuAD-Shifts
SQUAD-Shits (Amazon)
SQuAD-Shitts (Amazon)
DOD
DOD
500
SQuAD-Shifts (Reddit)
SQuAD-Shifts (Reddil)
SQuAD-Shifts (Reddil)
Accurecy Agreement
Acauvry Agreement
learayy Agreement
00D
14
05
10
SQuAD-Shifts (New York Times)
SQuAD--Shits (New York Times)
SQuAD-Shifts Shifts (New York Times)
Agreemant
Acoryyy
80-
Acounicy
Agreement
Agreementt
Acor
00D
000
000
SQuAD-Shifts Wik)
SQuAD-Shifts Wik)
SQuAD-Shitts Wiki
Acaunry Agreement
Agmemeent
Agrmement
000
000
000
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 36: ID vs OOD Fl and agreement of generative models finetuned on SQuAD from a single pretrained GPT2
44

---

A.6.2 Experiments starting from multiple foundation models
We also finetune base models of the GPT and OPT model families for generative question -answernnn using the same hyperparameters mentioned in the previous section. Similarly, we evaluate models using the FI score. Figure shows that AGL holds for all shifts in SQuAD- Shifts.. Furthermore, also holds for the mlqa-translatt translate--eeee test. es test split of the MLQA dataset English portion of Sinlish-Spanise translated MLQA questions.
32] which the
SQuAD-Shifts (Amazon)
SQuAD-Shits Times)
MLQA
3
SQuAD-Shifts (Reddit)
SQuAD-Shifts (New Wik))
Agreement
GPT
OPT
Figure 37: Generative models finetuned from different base models of the GPT and OPT family.
AGL holds for all SQuAD- splits and MLQA Spanish split.
45

---

A.7 Diversity under different learning rates and batch sizes
We test the robustness of our observation across learning rate and batch size for single base FM on CLIP embeddings. Figure 38 shows that AGL holds regardless of the learning rate or batch size used for fine-tuning, That is, AGL holds with random initialization of the head and does when for shuffling the data subset or randomizing the data ordering.
LR 1c-05, BS 100
LR 1e BS 200
LR 1e-05, BS 400
LR 1e-05, BS 800
Oreer Rit 285
Oreer R40.185
Orper NAU,30
Onder RiD 980
Subuer R20.004
R35-309
830,902
CHARIOC Pirelate
CITARI OC Pirelate
CITARLOC Porelate
*
-2
CHFRRIOC
CIFAR1O
CIFAR1O
CiFAR1O
CIFAR10
LR 0.0001, BS 100
LR 0.0001, BS 200
0.0001, BS 400
LR 0.0001, BS 800
H20, 190
Oveer R20.3
Oncer
CITARIOC Phoclate
CHTRR1CC Pirelate
CHARLOC Pioclate
CIFAR LDC Pheelatee
CiFAR10
CIFAR10
CiFAR10
CIFAR10
Subwer R20-070
Suboet 826.975
Subrer: 820,970
Sabaer 820.983
CITARIOC Phoclaa Pirclate
Ink: 1922
CIFARIDC Pirel late
Ink 923 199
CHESR1OC Pirclate
A22.978
CIFARJOC Preelate
IPR 423 999
Da To
12
CIFAR1O
So
19 21
St CIFAR1O
LR 100
0.01, BS
LR 0.01, BS 800
Ureer Nen or
Greer N20.884 Subee: Robos:R26:313
Ovder R2tLs
Order 820.309
Suboet R20-937
Subeet R22.075
Subert 820-375 arg 422 2922999
CITARIOC Pixelate
Irk: 420.980 988
CIRAPLOC Pioeliate ste
Irk 1966
A23.999
CIFARI2 LOC Pioelate
CHTARLOC +.
CIFAR10
CIFAR10
CIFAR10
CIFAR10
25
Figure 38: Over CLIP embeddings, test the diversity sources under different learning rates and batch sizes. While all models lie on the same ID versus OOD accuracy line (red), only the set of models trained with different random initialization (cyan) achieves ID versus OOD agreement with the same slope and bias as ID versus OOD accuracy.
46

---

A.8 Diversity Experiments for different PEFT methods
We compare full fine-tuning with different PEFT methods and observe that AGL holds for randomly initialized heads and does not for data ordering or data subsetting. Figure 39 shows single GPT2 trained with LoRA [25], IA3 [3], and BitFit [64]. The linear fit for the three PEFT methods aligns with full fine-tuning for randomly initialized heads.
SQuAD-Shifts (Reddit)
SQuAD-Shifts (Reddit)
5QuAD-Shifts (Reddit)
LoRA
LaRA
BitFit
BitFt
BRFit
Rin-Taninn
Fine-Tuning
Full Fine-Tuning
ACC
ACC
70
ACC
00D
00D
90
10
10-
ID
(a) Random Head
(b) Data Ordering
(c) Data Subsetting
Figure 39: ID (SQuAD) vs OOD (SQuAD-Shifts Reddit) Agreement trend from a single GPT2 for different PEFT methods (LoRA, IA3, BitFit). The accuracy line of all methods combined is shown in dotted gray while the agreement lines of each method is drawn separate colors.
47

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Sections 3
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long as clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
48

---

Justification: There is no theory to be proven other than the AGL phenomenon. The original work on AGL is cited throughout the entire paper. Specifically, the equations for calculating agreement can be found here: Appendix A.I.I
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix: A.2 Section
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
49

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [NA]
Justification: Our work has no algorithmic contributions and results can be replicated with https://githuu: caalbb.cookkeeee wwwaekk/grreeeeeennnne on-the-
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
ptapps.cc/pbllic/uuudee wttp///ooeSSssssiiooeaaaa
ssiiisolisyyyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Appendix: A.2 Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report MAE and MAPE throughout the paper. Specifically, they are in Tables 2
12
13
14
15
16
17
T8
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
50

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттоr rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: For all experiments we use most A6000's. For linear prohing, we use one RTX 8000. Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https http:://nnnuueeeeeeeeee httpe//wwww......ob.....cccccaaa
Answer: [Yes]
Justification:
We abide the code of ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work is on evaluating OOD performance of already existing foundation models.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
51

---

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes)
Justification:
We implement safeguards
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All models, datasets, and methods are cited.
Guidelines:
The answer NA means that the paper does not existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
52

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
53


Reviews:
[{"review_id": "jw3NskvxqY", "reviewer": "Reviewer_hhqs", "summary": "This submission studies the problem of predicting OOD performance given known in-domain performance. Building on top of recent work showing that ensembles can be used for this problem, by looking at agreements between components in the ensemble as surrogate labels to predict OOD performance, they find that a similar approach can be used for finetuned LLMs, as long as ensembles are generated by finetuning randomly initialized heads. The topic is timely, the paper is generally clearly written (although improvements are needed) and the empirical validation is good.", "questions": "If only the topmost layer is randomly initialized and trained, then all components of the ensemble should converge to the same parameter vector provided that they are trained for long enough because the optimization is a convex problem. Is the diversity due to the limited number of training step? If it is the case, I think the assumption should be made explicit.\n\nCould the authors compare the method of sec. 4 against the method of sec 3 directly?", "limitations": "no concern", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ relevant and important topic\n+ simple and practical approach", "weaknesses": "- some critical details are missing, for instance the authors should report in the main paper how to go from Acc, Agr to OOD (line 160-164).\n- the significance is unclear to me: It would be useful to predict OOD performance at larger scales as opposed for models that are finetuned for more steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions! We address each of your concerns below in detail. \n\n>__Some critical details are missing, for instance the authors should report in the main paper how to go from Acc, Agr to OOD (line 160-164).__\n\nWe apologize for the lack of clarity. The default prediction algorithm (ALineS) used to estimate OOD accuracy when agreement-on-the-line holds, is as follows:\n\n1. Say there is a linear trend in ID versus OOD agreement, i.e., for any two pairs of models $w, w’$ we have \n$\\Phi(Agr_{OOD}(w, w’)) = a * \\Phi(Agr_{ID}(w, w’)) + b$\n where $\\Phi(\\cdot)$ is the probit scaling. \n\n2. Estimate the slope and bias ($\\hat{a}, \\hat{b}$) of the above linear trend using OLS.\n\n3. Apply this linear transformation to accuracy:\n$\\Phi(Acc_{OOD}(w)) = a * \\Phi(Acc_{ID}(w)) + b$\n\nWe will update Section 2.3 to include equations defining accuracy-on-the-line and agreement-on-the-line in the main body. In addition, we have included a detailed discussion of the prediction algorithms (ALineS and ALineD) in Appendix A.1.1.\n\n>__The significance is unclear to me: It would be useful to predict OOD performance at larger scales as opposed for models that are finetuned for more steps.__\n\nThank you for this important question! You can utilize our results in the multiple base model section (Section 4) as a type of “scaling law”. In Figure 3, we showed that finetuned models of a large range of model scales (125M to 7B parameters) and model families (GPT, Llama2, etc.) observe the same ACL and AGL trends in QA and text classification. Using smaller models such as GPT2, we can compute the ACL/AGL trend, and use this to project the OOD performance of larger models _without any labels_ given their ID performance. \n\n>__If only the topmost layer is randomly initialized and trained, then all components of the ensemble should converge to the same parameter vector…because the optimization is a convex problem. Is the diversity due to the limited number of training step?__\n\nAs mentioned in Section 3, to construct the ensemble, we vary the number of epochs/training steps to get models with a wide range of ID accuracies. We will make this more clear in the revision - thank you for pointing this out! \n\n>__Could the authors compare the method of sec. 4 against the method of sec 3 directly?__\n\nThank you for this question! To reiterate, Section 3 studies AGL in ensembles of models finetuned from a _single_ base foundation model, where we establish that randomly initializing the head is important for observing AGL. In Section 4, we show that ensembles of models fine-tuned from different base foundation models (i.e. LLama, GPT, OPT) also exhibit AGL.\n\nTo answer your question, we do a direct comparison of AGL under these two scenarios. Specifically, we have a set of 14 finetuned GPT2 models, and we measure their agreement rate with\n\n*Setting 1:* other GPT2 models finetuned with different random initialization\n\n*Setting 2:* models finetuned from other base models (Llama, OPT).\n\nUsing these agreement rates, we predict the OOD performances of the GPT2 models using the ALine-S algorithm. We report the MAE and MAPE for the two settings below. Both are quite effective with very small MAE’s (estimates OOD accuracy with error less than 2%). \n\n__SQuAD vs SQuAD-Shifts Reddit__\n|          | Setting 1 | Setting 2|\n|--------------- |--------------|--------------|\n|MAE| 1.27| 1.54|\n|MAPE| 5.19| 4.82|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dt0sq0JHQp", "reviewer": "Reviewer_4CKp", "summary": "The authors of the paper propose a method to demonstrate that foundation models can exhibit agreement-on-the-line (AGL), under certain conditions. The existence of AGL can be used to predict the OOD capabilities of models without having access to the labels for the downstream tasks.", "questions": "I would like the authors to explain what the correlation $R$ in Section 5 refers to. If it is ID vs OOD agreement for various models, then it would mean that these should be nearly parallel to the $y = x$ line, which is a limitation of the setting where AGL is predictive of OOD performance.", "limitations": "I think the authors have adequately addressed the limitations of their work, and I cannot find any negative societal impact arising from their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- While AGL has been observed in the literature, the paper is novel (to the best of my knowledge) in applying it to the setting of foundation models.\n\n- The subject of the paper is also interesting - measuring the OOD performance of models is an important topic, and this paper does so with the added benefit of not explicitly requiring downstream labels (which, as the authors note, may be difficult to procure).\n\n- The experiments done by the authors are convincing, for the most part. The situations where foundation models exhibit AGL are clear, and it is easy to understand how the existence of AGL translates to good predictions about OOD performance.", "weaknesses": "- I believe that the paper's clarity, while overall good, could be improved further in certain points:\n\n  - Figure 1 should be a little clearer, with the caption being a little bit more detailed. This will help a lot with understanding of the key results of the paper, given the early position of the Figure in the document.\n\n  - I feel like the authors should elaborate a bit on lines 246 - 255, given that their result here is in contrast with previous statements made in the literature. A little more discussion here would be helpful.\n\n  - Similarly, Figure 3 should also be expanded a bit more, especially since in some settings the ID - OOD line lies directly on top of the $y= x$ axis, which I find very surprising.\n\n- The fact that AGL can predict OOD performance, while interesting, comes with the caveat in Section 5 that it cannot be currently applied to all datasets. The authors explicitly state that and provide a criterion to determine the setting in which AGL is predictive of OOD accuracy. Nevertheless, I think this Section requires a bit more detail (see also my question below).\n\nOverall, this is an interesting paper in my opinion, and I think my concerns with it currently are mostly based on the clarity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions! We address each of your concerns below in detail. \n\n>__Figure 1 should be a little clearer, with the caption being a little bit more detailed. This will help a lot with understanding of the key results of the paper…__\n\nThank you for the suggestion! We will update the Figure 1 caption if accepted to:  “The ID vs OOD lines for accuracy (yellow) and agreement (blue) for various datasets and fine-tuned ensembles. Each blue dot corresponds to a member of the ensemble and represents its ID (x) and OOD (y) accuracy; and each yellow dot corresponds to a pair of these members and represents their ID (x) and OOD (y) agreement. From CIFAR10 to CIFAR10C “Pixelate”' in linear probed CLIP, MNLI to SNLI in full fine-tuned OPT, and SQuAD to SQuAD-Shifts “Amazon” in full fine-tuned GPT2, we observe different agreement linear fits depending on the diversity source (columns) used to generate the ensemble.”\n\n>__I feel like the authors should elaborate a bit on lines 246 - 255, given that their result here is in contrast with previous statements made in the literature.__\n\nThank you for the suggestion! In our work, we demonstrated that _random initialization_ in particular is important to observe AGL in fine-tuned FM ensembles. We tie this finding to previous literature in lines 246-255 that study how well different sources of diversity in deep ensembles induce a related but different phenomena called GDE.  \n\nIn particular, GDE is a phenomenon in deep ensembles (neural networks, random forests) where ID accuracy tends to equal ID agreement exactly [1, 2]. Works have shown that, beyond the classical ensembling technique of bagging / data subsetting, deep ensembles induced by varying the data or model seed (ordering / initialization) can also induce this equality. \n\nOn the other hand, in our problem setting, we found that ensembles induced by different random initialization achieves AGL, while data ordering / subsetting cannot. Our setting is different from previous literature in two distinct ways:\n\n__1.__ AGL studies the _OOD_ agreement rate relative to their ID agreement, in contrast to the GDE phenomena which only regards the models’ ID agreement. We hypothesize that random initialization is much more important for observing the right levels of OOD agreement.\n\n__2.__ Models are only _lightly fine-tuned_ or linearly probed, unlike deep ensembles trained from scratch. Diversity sources may behave differently in this circumstance. \n\nWe will add this further discussion in the camera ready. \n\n[1] Jiang, et al. Assessing Generalization of SGD via Disagreement. ICLR 2022. \n\n[2] Nakkiran and Bansal. Distributional Generalization: A New Kind of Generalization. Preprint 2020. \n\n>__Figure 3 should also be expanded a bit more, especially since in some settings the ID - OOD line lies directly on top of the y=x axis, which I find very surprising.__\n\nThank you  for your suggestion! We will make sure to expand on the main takeaways from Fig 3. Notably, we observe that the agreement rate between models from different model families (e.g., between GPT and Llama models) observe agreement-on-the-line across different NLP tasks. \n\nIndeed, there are certain shifts such as from SQuAD to SQuAD-shifts New Wiki and SQuAD-shifts NYT where model performance barely drops. This is partially due to the large-scale pretraining, but also the distribution shift is simply much smaller – SQuAD was constructed using Wikipedia, so looks closer to SQuAD-Shifts New Wiki and NYT than Amazon reviews and Reddit [1].\n\nWhat is most peculiar is that ID vs OOD agreement tracks the linear trend of ID vs OOD accuracy accordingly. For small distribution shifts SQuAD-Shifts New Wiki and NYT, the models’ agreement ID vs OOD is also close to being y=x. For larger shifts such as SQuAD-shifts Amazon and Reddit, ID vs OOD agreement also moves away from the y=x line.\n\nWe will make these clarifications in Figure 3.\n\n[1] The Effect of Natural Distribution Shifts on Question Answering Models. Miller et al. 2020.\n\n>__I would like the authors to explain what the correlation R in Section 5 refers to. If it is ID vs OOD agreement for various models, then it would mean that these should be nearly parallel to the  y=x line…__\n\nThank you for the feedback! The $R^2$ we mention in Section 5 refers to the _standard linear regression coefficient of determination_. $R^2$ ranges between 0 and 1 and it measures how well the relationship between two variables, $X$ vs $Y$, can be explained by a linear function. When $R^2$ is high, $Y$ can be estimated as $aX + b$ with small residual error. In our paper, we use $R^2$ to measure the strength of the linear correlation in ID vs OOD agreement and ID vs OOD accuracy.  Note that $R^2$ is different from the slope of the linear fit. For example, consider SQuAD-shifts Reddit in Figure 3, both ID vs OOD accuracy and agreement have high $R^2$ values, but the slope is far away from the $y=x$ line.\n\nAccording to Baek et al. [1], $R^2$ can determine when AGL holds. In particular, when ID vs OOD agreement strictly follows a linear trend (i.e., the linear correlation has high $R^2 > 0.95$), then ACL also holds with the _same slope and bias_. In such circumstances, we can use AGL-based methods ALineS and ALineD to estimate the OOD performance of models precisely without labels. Interestingly, as we have demonstrated in our paper, many natural distribution shifts across image/text classification and QA observe AGL and ACL with high $R^2$. \n\nWe will make these points clearer in Section 5 of the final draft. \n\n[1] Baek et al. Agreement-on-the-line: Predicting the performance of neural networks under distribution shift. NeurIPS 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l4A5F4OBYG", "reviewer": "Reviewer_8Ly8", "summary": "The paper studies the applicability of agreement-on-the-line (AGL) to finetuned large models. Specifically, AGL is the phenomenon where the agreement in predictions of a collection of models on in-distribution (ID) data is linearly correlated with these models' agreement on out-of-distribution (OOD) data. This phenomenon is particularly interesting since earlier work has shown that this relation holds whenever the accuracy-on-the-line (ACL) holds; furthermore, these linear relation for agreement and accuracy is the same. While prior work has studied this phenomenon in a variety of settings, this paper studies instead this phenomenon for large finetuned models. The authors show that even in this regime, AGL holds in both vision and language settings. Furthermore, while prior work has shown that vision models pretrained on different distributions do not share the same AGL line, the authors find that the agreement of language models actually falls on the line.", "questions": "- The authors point out that they vary several factors when finetuning the large models, including the initialization of the linear head. I am a bit confused why different initialization would lead to substantially different accuracies. For example, in figures 1 and 2 (where there's supposed to be a single backbone that's finetuned several times), we can see that the x-axis values range from 10% to 90%. Can you please explain the reason behind this wide range?\n- Can you please add more results with different PEFT methods and analyze the difference (or the similarity) between them?", "limitations": "Yes", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written.\n- Generalizing the AGL phenomenon to finetuned large models is important as the use of these models becomes prevalent. \n- The authors run extensive evaluation on models of different sizes and on different downstream tasks.", "weaknesses": "- The paper lacks an analysis on the reason behind the different behavior observed between the vision and language models. Although these are two different modalities, I don't see a reason why vision and language models should behave differently. \n- The paper lacks an analysis of AGL in zero/few-shot settings.\n- The paper assumes that full finetuning and finetuning with LoRA lead to the same AGL phenomenon. While LoRA (and other PEFT methods) lead to a similar performance, their behavior might be different [1]. Such an analysis is needed, and the inclusion of more PEFT methods (QLoRA, BitFit, IA3, etc.) would be nice.\n\n\n[1] Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs. Pu et al. 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments and suggestions! We address each of your concerns below in detail. \n\n>__The paper lacks an analysis on the reason behind the different behavior observed between the vision and language models.__\n\nThe conclusions we make in this work apply to _both_ image and language models, we do not differentiate between these settings in any way. Models of both modalities require special attention to the source of diversity when fine-tuning from a common pretrained checkpoint to observe AGL. In particular, the diversity induced via random head initialization yields AGL, while the diversity induced via data reordering or data subsetting does not result in AGL. \n\nThe only differences we do make between vision and language tasks are \n\n__1)__ the fine-tuning strategy that we employ - e.g. full fine-tuning for question answering (since linear probing performs very poorly), and linear probing for image classification\n\n__2)__ the choice of evaluation metric (F1 score for QA and 0-1 accuracy for classification). \n\nAs emphasized in Section 2, our diversity results hold across tasks, regardless of what fine-tuning strategy or metric is employed. We will make this more clear in the next version. \n\n>__The paper lacks an analysis of AGL in zero/few-shot settings.__\n\nThanks for the feedback! We have some preliminary experiments on few-shot and zero-shot learning to answer your question in Figure 2 of our rebuttal PDF. We will add these results to the paper if you find that it strengthens our work.  \n\nWe consider few-shot linear probing over CLIP features, where we train on 10 examples in CIFAR10 per class and test models on the shift CIFAR10C Pixelate. Similar to fine-tuned FMs, we see that _by varying the random initialization of the linear probes, we can observe AGL and ACL._ On the other hand, data subsetting and data reordering trivially do not work in this setting as there’s only a small handful of training examples. \n\nThe zero-shot setting is a vastly different regime for understanding ACL and AGL because any downstream task is “out-of-distribution”. In future work, it may still be interesting to study the linear relationship between the pretraining loss (ID) versus the downstream task loss (OOD), or simply between two OOD tasks e.g., downstream task 1 versus downstream task 2. In Figure 2 of the rebuttal PDF, we take the intermediate pretraining checkpoints of OLMo 7B and evaluate their zero-shot performance on SQuAD versus SQuAD-shifts Reddit. On the contrary to finetuned models, we see that both ACL and AGL do not hold in this setting. \n\n>__The paper assumes that full finetuning and finetuning with LoRA lead to the same AGL phenomenon. While LoRA (and other PEFT methods) lead to a similar performance, their behavior might be different [1]. Such an analysis is needed, and the inclusion of more PEFT methods (QLoRA, BitFit, IA3, etc.) would be nice.__\n\t\t\t\t\t\t    \n>__Can you please add more results with different PEFT methods and analyze the difference (or the similarity) between them?__\n\nThank you for the suggestion! We first want to clarify that we do not claim that PEFT and full-finetuned models behave the same generally or that they always lie on the same ACL and AGL trend. For example, other works have reported circumstances where PEFT and full-finetuned models observe different levels of effective robustness under distribution shift (i.e., different ACL slopes) [1]. The reason we group these two finetuning methods together in our work is mostly for notational convenience (i.e., LP versus FFT) since across the datasets we evaluate in our work, PEFT and full-finetuned models do observe the same effective robustness. We will make this more clear in Section 2. \n\nTo further strengthen our work, we have included additional experiments that directly compare different PEFT methods. We trained GPT2 for random head initialization, data ordering, and data subsetting with LoRA, IA3, and BitFit as shown in Figure 1 of the rebuttal PDF. Regardless of the PEFT method, the accuracy points lie on the same line and AGL holds best with random head initialization. Interestingly, this indicates that even with different PEFT methods, we can observe AGL in ensembles as long as the linear head is randomly initialized.\n\n[1] Chen, et al. Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Models. Neurips 2023. \n\n\n>__I am a bit confused why different initialization would lead to substantially different accuracies. For example, in figures 1 and 2…we can see that the x-axis values range from 10% to 90%. Can you please explain the reason behind this wide range?__\n\nGood question! The large range of accuracies is a consequence of models being trained for varying amounts of training epochs and this is necessary to observe the full ACL/AGL linear trends. For each source of diversity we also vary the number of epochs (not train until convergence) to obtain this range of accuracies. As we finetune from a randomly initialized linear head, the model performance at the beginning of finetuning is almost random (10% for CIFAR10 classification).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ebEgoWiddV", "reviewer": "Reviewer_H4MK", "summary": "The paper shows that agreement-on-the-line holds in finetuned foundation models across vision and language benchmarks and finds that random head initialization is critical for the phenomena. The authors also show that agreement-on-the-line holds in ensembles of different pretrained models. They demonstrate usage of agreement-on-the-line to predict OOD performance.", "questions": "1.\tIf you only train a linear head and freeze all the other parameters, isn’t the problem convex? Then, suppose training to convergence, initialization could potentially bias the final solution. However, this bias should be a result from the random initialization in the data kernel space. Why would this type of randomness be meaningful at all? \n2.\tWhat do you think are potential causes for the larger slopes of AGL compared to ACL if trained with data ordering or data subsetting?", "limitations": "The authors have listed limitations at the end. However, I think one major limitation is when the correlation coefficient is low, other methods will outperform AGL, which is shown in Table 5. I suggest authors include this limitation.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper is overall clear and well-written with substantial and thorough experiments on phenomenon of agreement-on-the-line across various language and vision benchmarks. The question of understanding and predicting model performance under distribution shifts is important in the field. The authors show that the proposed method of using agreement-on-the-line to predict OOD performance outperforms previous methods in most cases.", "weaknesses": "Minor points:\n1.\tThe panel labels in Fig. 31 and 32 has a repetition of Random Head, which should be corrected.\n2.\tThe authors fail to discuss the robustness of the method. Specifically, it is important to know how robust the method is in terms of changing different hyperparameters.\n\nMajor points:\n1.\tNovelty: the authors extend a previously proposed method (Baek et al. 2022) to finetuned foundation models. Although the aspect that observing AGL holds in this new setup is to some extent is novel, the methodology itself lacks novelty. \n2.\tThe paper is mostly experimental results without providing any insights into why one would expect AGL to hold or correlate with ACL. Without any explanation or theoretical guarantee, it is debatable whether one can trust this method with new models or new data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">__The panel labels in Fig. 31 and 32 has a repetition of Random Head, which should be corrected.__\n\nThank you for catching this! The third panel (c) corresponds with Data Subsetting, not Random Head. We will correct this in the next version. \n\n>__…it is important to know how robust the method is in terms of changing different hyperparameters.__\n\nThank you for your feedback! None of the hyperparameters we use to linear probe or fully-finetune the models in this work have been picked to observe this phenomena. In Figure 3 in our rebuttal PDF, we see that across different learning rates and batch sizes, linear models over CLIP only observe AGL by varying the random initialization.\n\n>__Novelty: the authors extend a previously proposed method (Baek et al. 2022) to finetuned foundation models. Although the aspect that observing AGL holds in this new setup is to some extent is novel, the methodology itself lacks novelty.__\n\nThe novelty of our work stems from several factors. \n\n*First, OOD performance estimation of FMs is a critical problem.* Foundation models are utilized for a variety of downstream tasks and to safely use them, it is important that we have ways to estimate their OOD performance, especially in limited label settings. This problem is largely underexplored and we still lack an understanding of when older methods transfer.\n\n*Second, before this work, it was unintuitive whether AGL holds in FMs.* The importance of diversity source in finetuned FMs, in contrast to deep ensembles trained from scratch, is quite surprising. This has interesting theoretical implications. \n\n 1. Finetuning on top of pretrained weights can produce models with surprising levels of decorrelated errors when trained from different randomly initialized heads. \n\n 2. AGL can hold in linear probes, in contrast to Baek et al. which argue AGL only occurs in neural networks. \n\nWe believe these findings would be interesting to both practitioners and theoreticians. \n\n*Third, our method can effectively estimate OOD performance for a wider range of scenarios than other baselines, which break for non-classification tasks.*  In QA benchmarks, the prediction error is 20% smaller than popular confidence-based prediction methods.\n\n>__The paper is mostly experimental results…Without any explanation or theoretical guarantee, it is debatable whether one can trust this method with new models or new data.__\n\nThank you for the feedback! While we do not provide exact theoretical guarantees, the conclusions we make about ensemble diversity in FMs and their effect on observing AGL/ACL hold across hundreds of finetuned models we tested from different model familes (GPT, OPT, Llama), 50+ distribution benchmarks (Appendix A.3 and A.4), hyperparameters (see learning rate/batch size sweep in rebuttal Figure 3), finetuning strategies (LP, LoRA, full finetuning; other PEFT methods in rebuttal Figure 1). We hope that our rigorous experimental report can demonstrate that ACL/AGL is a powerful tool for predicting the performance of FMs. \n\nFurthermore, the coefficient of determination $R^2$ of ID vs OOD agreement (see Sec. 5) gives practitioners a rough guarantee for when the performance estimate derived by AGL is reliable. This is immensely useful in comparison to other confidence-based baseline methods which also do not come with any formal guarantees. \n\nWe’d also like to point to several theoretical works that have tried to characterize when ACL and AGL hold [1, 2]. We believe our study on the importance of random initialization for observing AGL introduces a new conceptual angle for understanding deep ensembles and what actually induces AGL. We hope this can inspire further theoretical research on this important topic. \n\n[1] Mania and Sra. Why do classifier accuracies show linear trends under distribution shift? 2020. \n\n[2] Lee, et al. Demystifying Disagreement-on-the-Line in High Dimensions. 2023. \n\n>__If you only train a linear head, isn’t the problem convex? Then, suppose training to convergence,…(bias) result from the random initialization in the data kernel space. Why would this type of randomness be meaningful?__\n\n>__What are potential causes for the larger slopes of AGL compared to ACL if trained with data ordering or data subsetting?__\n\nThank you for the great question! In all of our ID versus OOD scatter plots, you will see that we have models of a wide range of accuracies. This is because in addition to the diversity source, these models are also _finetuned for different timesteps_. In fact, a lot of these “partially” trained models can be close to their initialization, and any bias from random initialization isn’t restricted to the kernel of the training data. However, the bias in the kernel induced by random initialization may be important for achieving the low _OOD_ agreement rates. We provide some intuition below. \n\nConsider the linear probe setting wherein we train a linear classifier on top of fixed features. Suppose we assume that the ID training data is low rank. Let S be the span of the ID training data, and N be the corresponding null-space that has some overlap with the OOD data. The training iterations only update the classifier in S. As a result, under different random initializations, classifiers would retain their different initializations on S and lead to lower OOD agreement. However, if the initializations are fixed, and only the data ordering or subset varies, these orthogonal components no longer vary, leading to higher OOD agreement. Intuitively, such an argument should carry over to heavy-tailed training data as well, beyond low-rank. We were unable to come up with an analogous interpretable diversity introduced by data-subsetting or ordering. \n\nOverall, the phenomenon of agreement-on-the-line is poorly understood, and we think our work reveals new empirical observations that help form a better picture. We believe our work would inspire and inform future work that rigorously and comprehensively explains AGL.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their great feedback and questions about our paper! The reviewers generally found our paper interesting and praised our work for proposing a simple yet effective solution to predict the OOD performance of FMs. Here, we briefly summarize the common concerns, and new experiments we’ve added in our rebuttal PDF: 

__*How do you achieve a wide range of accuracies and agreements in the linear probing setting where the loss landscape is convex?*__ In each ensemble, we also vary the number of epochs we finetune each model. Regardless, the random initialization also has to vary to observe AGL. 

__*Can we use this method to forecast OOD performance with larger scales?*__ Yes! We showed in Section 5 that on many language tasks, foundation models from different families (GPT, OPT, LLama) of different sizes all lie on the same ACL and AGL trends. This means we can estimate the linear trend using smaller models to extrapolate the performance of larger models with no labels OOD. 

__*Is it interesting to study fine-tuned FMs?*__  It is common practice to finetune FMs by linear probing or LoRA, and we believe our study can apply to many practical use cases for OOD estimation. As requested, we also extend our study to few-shot and zero-shot settings in rebuttal Figure 2. In the few-shot setting, we also observe that ACL/AGL hold in CLIP by varying the random initialization. On the other hand, zero-shot language models do not necessarily observe AGL/ACL trends as strongly as fine-tuned models on SQuAD versus SQuAD-Shifts. Zero-shot models may behave differently as neither SQuAD nor SQuAD-Shifts is “in-distribution”. 

__*Are there any theoretical guarantees we provide about ACL or AGL?*__ While we do not provide exact theoretical guarantees, the conclusions we make about ensemble diversity in fine-tuned FMs and their effect on observing AGL/ACL hold across hundreds of fine-tuned models we tested from different model families (GPT, OPT, Llama), 50+ distribution benchmarks (Appendix A.3 and A.4), hyperparameters (see learning rate/batch size sweep in rebuttal Figure 3), fine-tuning strategies (LP, LoRA, full fine-tuning; other PEFT methods in rebuttal Figure 1). We hope that our rigorous experimental report can demonstrate that ACL/AGL is a powerful tool for predicting the performance of FMs.

Meta-review: This paper investigates the applicability of agreement-on-the-line (AGL) to finetuned large language models for predicting out-of-distribution (OOD) performance. Reviewers generally found the paper to be well-written and technically sound, the method is simple and practical, and importantly, the empirical evaluation is extensive and done across various benchmarks and model sizes ("The authors run extensive evaluation on models of different sizes and on different downstream tasks." - 8Ly8). Reviewers raised few questions regarding the clarity of figures and explanations, or additional studies of AGL in the zero/few-shot settings and with different PEFT methods, which appears to be sufficiently addressed during the rebuttal stage. However, there is lingering concern regarding the lack of theoretical insight or sufficiently convincing explanation on the success of AGL ("Without any explanations or theoretical insights, I am doubtful whether AGL (particularly given its origin from random initialization in a convex problem) is indeed meaningful or just a mere coincidence that has trivial explanation" - H4MK), which authors may consider addressing during paper revision stage.
Decision: Accept (poster)