Predicting the Performance of Foundation Models via Agreement-on-the-Line

Authors: rahul  saxena
taeyoun  kim
aman  mehra
christina  baek
zico  kolter
aditi  raghunathan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 01:08:01

Abstract: Estimating the out-of-distribution performance in regimes where labels are scarce is critical to safely deploy foundation models. Recently, it was shown that ensembles of neural networks observe the phenomena "agreement-on-the-line", which can be leveraged to reliably predict OOD performance without labels. However, in contrast to classical neural networks that are trained on in-distribution data from scratch for numerous epochs, foundation models undergo minimal finetuning from heavily pretrained weights, which may reduce the ensemble diversity needed to observe agreement-on-the-line. In our work, we demonstrate that when lightly finetuning multiple runs from a $\textit{single}$ foundation model, the choice of randomness during training (linear head initialization, data ordering, and data subsetting) can lead to drastically different levels of agreement-on-the-line in the resulting ensemble. Surprisingly, only random head initialization is able to reliably induce agreement-on-the-line in finetuned foundation models across vision and language benchmarks. Second, we demonstrate that ensembles of $\textit{multiple}$ foundation models pretrained on different datasets but finetuned on the same task can also show agreement-on-the-line. In total, by careful construction of a diverse ensemble, we can utilize agreement-on-the-line-based methods to predict the OOD performance of foundation models with high precision.

Main paper: 
 introduction: 
 Foundation models (FM), or large models first pretrained on open world data then finetuned or prompted for a specific downstream task, have proven to be powerful solutions for many common machine learning problems. A notable trait about FMs is that they are far more robust to distribution shift than other deep learning approaches -across image and language benchmarks, they suffer a smaller performance degradation on out-of-distribution (OOD) data, that may vary substantially from the in-distribution (ID) finetuning data [43, 42, 7, 60, 58, 14] . From clinical decision-making in different hospitals to navigating robots through unseen terrains, FMs are increasingly utilized for tasks prone to distribution shift. However, evaluating these models in OOD settings remains difficult: in many cases, acquiring labels for OOD data is costly and inefficient, while unlabled OOD data is much easier to collect. Although the field has explored other means for estimating OOD accuracy without labeled data, they are not ideal for FMs. A reliable FM performance estimator has the following desirable properties. First, the method must be computationally efficient to account for FMs' large model size. Second, FMs are leveraged for many different tasks (e.g., classification, question-answering, regression), so the method should also be versatile across tasks. Third, as we will see, methods for finetuned FMs may require different model assumptions from neural networks trained from scratch.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Figure 1 : The ID vs OOD lines for accuracy (orange) and agreement (blue) for various datasets and fine-tuned ensembles. Each blue dot corresponds to a member of the ensemble and represents the ID (x) and OOD (y) accuracy. Each orange dot corresponds to a pair of these members and represents the ID (x) and OOD (y) agreement. From CIFAR10 to CIFAR10C "Pixelate" in linear probed CLIP, MNLI to SNLI in full fine-tuned OPT, and SQuAD to SQuAD-Shifts "Amazon" in full fine-tuned GPT2, we observe that randomly initializing the head as the diversity source for generating ensembles (columns) shows the closest agreement linear fit to accuracy.Recently, [2] proposed a promising method for estimating the OOD accuracy of deep networks using the agreement between pairs of these classifiers (i.e., how often two classifiers make the same prediction). For distribution shifts where models observe a strong linear correlation in ID versus OOD accuracy -a common phenomenon in vision and language benchmarks [39, 1] -a strong linear correlation also holds for ID versus OOD agreement with extremely similar slopes and intercepts. These effects are referred to as accuracy-on-the-line (ACL) and agreement-on-the-line (AGL) respectively, and together they provide a simple method for estimating OOD accuracy via unlabeled data alone. Namely, without any OOD labels, we can instead measure the linear fit of ID versus OOD agreement as a proxy for the linear fit of ID versus OOD accuracy. With this linear fit, we can verify whether ACL holds by using the correlation strength of agreement's linear trend and estimating each model's OOD accuracy by linearly transforming ID accuracy. This simple approach has shown to reliably predict the OOD accuracy of models within a few percentage points across classification and question-answering tasks.Unfortunately, while the method has several practical advantages, it is unclear whether finetuned FMs also observe the necessary AGL phenomena. Intuitively, a prerequisite to observing AGL is a diverse ensemble of classifiers. Since OOD accuracy falls below ID accuracy, if the linear trend in ID versus OOD agreement is to match that of accuracy, models must also agree much less OOD than ID. For this to happen, errors between any two models must be sufficiently decorrelated. [2] observes AGL in ensembles of neural networks trained for hundreds of epochs from scratch where it is conceivable that the stochasticity between training runs leads to large divergences in the weight space, and corresponding models have diverse OOD predictions. However, in the case of finetuned FMs, models are much closer in the weight space. FMs are often either linear probed over the same pretrained weights or full finetuned for a few epochs with a small learning rate and intuitively, such light finetuning may lead to models that "revert back" to their pretrained behavior to make highly correlated predictions OOD.This raises the question: can we enforce AGL in this paradigm of lightly finetuning heavily pretrained models? In this work, we conduct an extensive study across several modalities, e.g., CLIP-based image classification and LLM-based question-answering, and training regiments, e.g., full finetuning and linear probing, to understand when AGL holds for finetuned FMs. We first investigate whether AGL appears in an ensemble of finetuned models from a single base FM. To collect a deep ensemble, the following sources of diversity can be injected into the finetuning process: 1) random initialization of the linear head; 2) random data ordering; and 3) random data subsetting. We find that not every source of diversity during fine-tuning on ID data manifests in sufficient diversity OOD, breaking the matching linear fits in ID versus OOD accuracy and agreement. Interestingly, finetuning models from different random initializations of the linear head consistently induces AGL across benchmarks. In contrast, neural networks trained from scratch observe AGL irrespective to these diversity sources.Second, we show that finetuned models from multiple different base FMs can be leveraged for AGLbased performance estimation. As base FMs can be pretrained with different datasets, architectures, and training regiments, the linear trends in ID versus OOD accuracy and agreement may break altogether in such ensembles. Indeed previous works indicate that on vision tasks, FMs pretrained on different image corpora can have different levels of OOD robustness for the same ID performance [17, 43, 51] . On the contrary, we find that on language tasks, FMs pretrained on different text corpora observe both AGL and ACL across question-answering and text classification tasks.In total, we develop simple techniques for applying AGL-based performance estimation methods to predict the OOD performance of foundation models. We demonstrate that the AGL phenomenon is not limited to ensembles of neural networks trained from scratch. By simply finetuning FMs from random initializations of the linear head, we can observe the phenomena in FMs across a wide variety of tasks (classification, question-answering) and modalities (vision, language) and training procedures (linear probing, full finetuning). We find that AGL is the only method to accurately estimate the performance of finetuned FMs across all tasks, surpassing other performance estimation baselines by a significant margin as large as 20% mean absolute percentage error.2 Background and related work 
setup: 
 We are interested in evaluating models that map an input x ∈ X to a discrete output y ∈ Y. In particular, we finetune foundation models. For a base model B, let f (B) denote a finetuned version of B. In this work, we consider a variety of foundation models: GPT2 [42] , OPT [65] , Llama2 [53] , BERT [14] , and CLIP [43] .Finetuning strategies. We have access to labeled data from some distribution D ID that we use for obtaining f (B) from B. In this work, we consider the following standard finetuning procedures.1. Linear probing (LP): Given features from the base model B θ , we train a linear head v such that the final classifier maps the score v ⊤ B θ (x) to a predicted class. We randomly initialize v and update v via gradient steps on a suitable loss function. The base model parameters remain frozen. We refer to v as either a linear probe (classification), or span prediction head (question-answering) depending on the task.We update all parameters of the backbone B θ and the linear head v using a small learning rate. When infeasible to update all parameters, we perform low-rank adaptation (LoRA) [25] to reduce the number of trainable parameters while still effectively updating the feature extractor B θ . In this work, we do not distinguish between LoRA and FFT as they conceptually achieve the same effect, and seem to show similar empirical trends in our studies.OOD performance estimation. Given access to a labeled validation set from D ID and unlabeled samples from a related but different distribution D OOD , our goal is to estimate performance on D OOD . We consider the standard performance metrics for various tasks: Accuracy ℓ 0-1 : Y → [0, 1] for classification, and Exact Match ℓ EM : Y → [0, 1] and Macro-averaged F1 score ℓ F1 : Y → [0, 1] for question-answering. We use ℓ to denote the appropriate metric in the context.There is rich literature on OOD performance estimation for deep networks, with a variety of proposed approaches. Initial works focused on upper bounding the degree of distribution shift through data and/or model dependent metrics, e.g., uniform convergence bounds using H-divergence [4, 37, 11, 30] . However, these bounds tend to be loose for deep networks [39] . The following works try to estimate the performance exactly.For classification, [23, 22, 19, 16, 21] leverage the model's confidence to predict the OOD performance. Since deep models are typically overconfident, these models are first calibrated in-distribution by temperature scaling. Similar methods are uncertainty quantification works that directly calibrate models under distribution shift [62, 67, 41] . Confidence based methods are commonly utilized in practice, and favorable for foundation models as they are computationally light and model-agnostic. However, they often fail for large shifts [19] and are often well-defined for accuracy but not other common metrics like F1 score. These can be limiting factors for foundation models which are applied to a broad array of tasks. Still, as they are the most common estimation methods, we utilize them as the baselines in our work. [49, 12, 13 ] also measure model behavior on known auxiliary tasks to understand model behavior under the distribution shift at hand. However, these approaches tend to be overfit to specific datasets or modalities. Similar to AGL, there are prediction methods that utilize information from ensembles. Oftentimes a separate "reference" ensemble is trained on some objective to predict the performance of a "target" model [10, 63, 8] . These methods have a higher computational cost than AGL. Although AGL also requires at least 3 models to compute agreement, these models only undergo generic finetuning. Thus, it is a better suited approach for evaluating foundation models, especially if off-the-shelf finetuned models are readily available, e.g., from Huggingface (see Section 4).Overall, there is growing attention towards understanding the safety and reliability of foundation models. To understand the effective robustness of FMs under distribution shift, recent works focus on studying the "accuracy-on-the-line" phenomena [39] (details in next subsection) and designing benchmarks that expose different failure modes of large models [36, 54] . However, unsupervised OOD performance estimation is underexplored in this modern setting, in terms of new methods and the transferability of old methods to large pretrained models.We are interested in adapting the method "agreement-on-the-line" (AGL) [2] for OOD estimation as it obtains state-of-the-art performance estimation across several distribution shifts. AGL is based on an earlier observation called "accuracy-on-the-line" (ACL) -across common distribution shift benchmarks, there is a strong linear correlation between the ID and OOD performance of models [39, 45-47, 61, 51, 38] . ACL can also be observed in FMs for image classification, e.g., CIFAR10C [22] , ImageNetV2 [46] , FMoW-WILDS [28] , and question-answering, e.g., SQuAD-Shifts [38] . However, ACL does not always hold, e.g., Camelyon-WILDS [39] and SearchQA [1] .While ACL is a striking phenomenon, it does not immediately provide a practical method to estimate OOD performance-computing the linear fit of ID versus OOD accuracy requires labeled samples from D OOD . Alternatively, we can estimate this linear trend exactly using only the agreement between neural networks [2] . Formally, given a pair of models f 1 and f 2 that map inputs to labels, accuracy and agreement is defined asEQUATIONwhere ℓ is the appropriate performance metric of interest. While accuracy requires access to the ground truth labels y, agreement only requires access to unlabeled data and a pair of models. [2] observes that when ID versus OOD accuracy is strongly linearly correlated between neural networks, i.e., ACL, then the ID versus OOD agreement of pairs of these models also observe a strong linear correlation with the same linear slope and bias. Furthermore, when accuracies do not show a linear correlation, agreements also do not. This coupled phenomenon is dubbed "agreement-on-the-line" (AGL).To use AGL for OOD performance estimation, one may obtain the slope and bias of the agreement line with unlabeled data, and then estimate the OOD performance by linearly transforming the ID validation performance. Specifically, with a collection of models F = {f 1 , f 2 , ..., f n }, AGL suggests that ID versus OOD accuracy observe a strong linear correlation if and only if ID versus OOD agreement observes a strong linear correlation and when they do, the slopes and biases match:EQUATIONΦ -1 is the probit transform used to induce a better linear fit as used in [2, 39] . Provided access to Acc ID (f i ), Agr ID (f i , f j ), Agr OOD (f i , f j ) ∀i, j, we can estimate Acc OOD (f i ) for all f i ∈ F. We refer the reader to [2] for formal AGL-based performance estimation algorithms (ALine-S and ALine-D), which we also provide in Appendix A.1.1.We present the same training data to each model but shuffle the order of the data, i.e., model observes different minibatches.3. Data subsetting We i.i.d. sample p% subset of the data to train over. In the main body, we report models trained on independently sampled 10% of the training data, other proportions of 30% and 50% are reported in Appendix A.4.We perturb one source of diversity at a time and study whether AGL occurs in each resulting model ensemble. For each setting, we also vary the number of training epochs to collect models with different ID performance, which is necessary to obtain a meaningful linear correlation in accuracy.The additional randomness induced by different training epochs does not affect the observations we make. We use at most four A6000's for all experiments except for linear probing where we use one RTX 8000. 
predicting ood performance: single base foundation model: 
 We first evaluate whether AGL appears in an ensemble of multiple finetuned runs of a single base foundation model. This would enable precise OOD performance estimates for each ensemble member.A practitioner may naively gather a finetuned ensemble by training a couple runs with different seeds or hyperparameters. However, an overriding concern is that even with some randomness in the finetuning process, linear probing or light full-finetuning over the same base model may lead to solutions with very correlated predictions. We extensively evaluate the following methods of introducing diversity into the finetuning process to see what approach (if any) can lead to AGL.1. Random linear heads We initialize the last layer of the network (i.e., the linear head) randomly, instead of via some zero-shot or pre-specified manner.We first investigate the effect of diversity source on AGL behavior for vision benchmarks. For image classification, a common pipeline is to finetune over a CLIP [43] pretrained foundation model.We finetune over OpenCLIP ViT-B/32 model trained on LAION-2B [26] .Given its well-established zero-shot capabilities, a popular method of finetuning CLIP is to simply employ linear probing on top of the CLIP representation. We take particular interest in evaluating the OOD performance of an ensemble of linear models trained on top of frozen base model representations.Datasets We evaluate ensembles on synthetic corruptions (CIFAR10C, CIFAR100C, ImageNetC), dataset replication shifts (CIFAR10.1, ImageNetV2), style shifts (OfficeHome), geographical and temporal shifts (FMoW-WILDS, iWildCam-WILDS), and interlaboratory shifts in medicine (Camelyon17-WILDS). iWildCam-WILDS exhibits weak ACL and Camelyon17-WILDS doesn't exhibit any ACL [39] . We test on iWildCam-WILDS and Camelyon17-WILDS to verify AGL's negative condition, i.e., when the linear correlation does not exist in ID versus OOD accuracy, it also does not exist in agreement.Table 1 : We evaluate models on the following distribution shift benchmarks.ID OOD CIFAR10 [29] CIFAR10C [22] , CIFAR10.1 [45] CIFAR100 [29] CIFAR100C [22] ImageNet [48] ImageNetC [22] , CIFAR10.1 [45] FMoW ID [28] FMoW OOD [28] iWildCam ID [28] iWildCam OOD [28] Camelyon17 ID [28] Camelyon17 OOD [28] OfficeHome [56] All (ID, OOD) pairings of domains Art, ClipArt, Product, Real World MNLI [59] MNLI-Mismatched [59] , SNLI [6] SQuAD [44] SQuAD-Shifts [38] Camelyon17-WILDS (OOD)Figure 2 : In ensembles with diverse random initializations, ACL and AGL holds across benchmarks in linear probed CLIP models. Similar to [2] , neither ACL nor AGL holds for the Camelyon17-WILDS Results Across vision benchmarks, linear probed CLIP models observe ACL, i.e. there is a strong linear correlation in the ID versus OOD performance. Similarly, on the same datasets, we can observe a corresponding strong linear correlation in agreement across ensembles injected with diversity in linear head initialization, data ordering, and data subsetting (Figures 1 ). However, we find that only ensembles with diverse initialization leads to AGL where the linear trend of agreement and accuracy have a matching slope and bias (Figure 2 ). See Appendix A.3.2 for results on other datasets. In model ensembles obtained by data ordering and data subsetting, we observe a consistent trend where the agreement trend observe a much higher slope close to the diagonal y = x line. These results are not specific to linear probing alone. In full finetuned CLIP models, we also observe that random linear heads induce the most reliable AGL behavior (See Appendix A.3). Note that this setting is still notably different from [2] where models are heavily trained for tens to hundreds of epochs often with a large learning rate, which causes AGL behavior to be more robust to the source of diversity used to induce the ensemble (See Appendix A.3.4).We conduct a similar systematic investigation of AGL in finetuned runs of a single base language model. Similar to CLIP linear probing, we find that AGL cannot be observed without random head initialization in language models evaluated on text classification and extractive question-answering tasks. While we mostly focus on tasks that require a linear head during finetuning, we also conduct a diversity study on generative tasks where the base model is finetuned directly in Appendix A.6.1. Full Finetuned Language Models We evaluate over a collection of 450 full finetuned runs of several base FMs: GPT2-Medium [42] and OPT-125M [65] . Models are full finetuned for up to 20 epochs with a small learning rate (≤ 1e -6 ). Hyperparameters specifics can be found in Appendix A.2. We do not conduct a linear probing study for question-answering as it leads to poorly performing models. For text classification, we also conduct a linear probing study in Appendix A.5.We test models on a text classification shift from MNLI [59] in the GLUE benchmark [57] to MNLI-Mismatched [59] and SNLI [6] . We also evaluate extractive question-answering models on the shift from SQuAD v1.1 [44] to SQuAD-Shifts [38] .We evaluate models on accuracy for text classification and F1 score for question-answering. Similar to our findings in CLIP, in both text classification and question-answering benchmarks, ensembles of full finetuned LLMs observe AGL when models are trained from different randomly initialized linear or span heads while data ordering and data subsetting observe an agreement trend closer to the diagonal y = x line (Figure 1 and Appendix A.5). We note that with full finetuning, the differences in AGL behavior between diversity sources are not as stark as with linearly probed models. In some sense, how model diversity is achieved becomes increasingly less important for observing AGL as the base model parameters also diverge, with ensembles of models heavily trained from scratch at the extreme [2] .Across image and language modalities, we demonstrate that ensembles of finetuned FMs can also observe agreement-on-the-line similar to heavily trained CNN's [2] . In both domains and regardless of the fine-tuning strategy, e.g. FFT and LP, or metric, e.g. F1 and Accuracy, employed, the diversity induced via random head initialization yields AGL, while the diversity induced via data reordering or data subsetting does not. This phenomenon can be observed across hyperparameters (Appendix A.7) and different PEFT methods (Appendix A.8). With a single heavily pre-trained base FM, one may think that light finetuning leads to downstream models with highly correlated behavior under distribution shift. However, simply randomly initializing the linear head alone induces sufficiently decorrelated models for observing AGL. The diversity in the ensemble becomes important when predicting the OOD performance of models using downstream AGL-based methods. In Table 2 , we show that AGL-based methods can only accurately predict the OOD performance of models in ensembles with diverse initialization, and cannot with data subsetting or ordering.Furthermore, our findings contrast previous work that suggest AGL is a neural-network specific phenomenon [2, 31] , unlike ACL which is model agnostic [39] . Specifically, [2] report that linear models trained on top of the flattened CIFAR10 images do not observe AGL. However, we find that, on top of CLIP features, linear models can exhibit AGL with random initialization. Previous work on the Generalization Disagreement Equality [27] contend that data subsetting leads to the most diversity in model predictions for deep ensembles (neural networks, random forests). Specifically, in-distribution, the agreement rate between pairs of models was shown to equal their expected accuracy in ensembles obtained by data subsetting, while those that vary random initialization has slightly higher agreement [27, 40] . On the other hand, in our problem setting of out-of-distribution datasets on FMs, we found that ensembles induced by different random initialization achieves AGL, while data ordering or subsetting cannot. Our setting is different from previous literature in two ways: (1) AGL studies the OOD agreement rate relative to their ID agreement, in contrast to the GDE phenomenon which only regards the models' ID agreement. We hypothesize that random initialization is much more important for observing the right levels of OOD agreement. (2) Models 4 Predicting OOD performance: multiple foundation modelsAlternatively, we consider ensembling multiple base foundation models. First, ACL may not hold because the base models are heavily pretrained on different data corpuses. This may cause respective downstream models to have different ID versus OOD accuracy trends or "effective robustness" [17] . On vision tasks, for example, linear probing over CLIP, EfficientNet [50] , ViT [15] , and BYOL [20] observe varying robustness trends [43] . Second, even when ACL does hold, it is unclear whether the ensembles will also observe AGL. Here the problem is different from the single base model setting: any pair of foundation models finetuned from different base models may agree too little, or OOD agreement rate may vary across model pairs depending on the similarity of the pretraining corpus, breaking the linear correlation of agreement entirely. Yet, we observe that for language models and tasks, ensembles of finetuned FMs from a wide range of base models observe both ACL and AGL.Models We finetune models from OPT-125M, OPT-350M, OPT-1.3B [65] , GPT2, GPT2-Medium, GPT2-Large, GPT2-XL [42] , GPT-Neo-135M [5] , Llama2-7B [53] , Alpaca-7B [52] , and Vicuna-7B [9] . We fully finetune OPT and GPT models and LoRA finetune Llama, Alpaca, and Vicuna. These models are pretrained on different mixtures of BookCorpus [66] , Stories [55] , PILE [18] , CCNews v2 corpus, and PushShift.io Reddit [3] . Alpaca and Vicuna are instruction-finetuned over Llama2. 
results: 
 We investigate the AGL behavior of an ensemble of foundation models finetuned from diverse base models in Figure 3 for question-answering. First note that base LLM models pretrained on different text corpora lead to finetuned models that lie on the same linear trend in accuracy. Unlike the different accuracy trends observed by different vision foundation models [42] , we suspect that the pretraining datasets for the language models in our study observe much more homogeneity. Second, the ID versus OOD agreement between pairs of models in this ensemble, including those between different base foundation models, is also strongly correlated and the slope and intercept closely matches that of accuracy. In other words, ensembles of different base models also observe AGL without any special regularization for ensemble diversity. The same holds for generative QA tasks (Appendix A.6.2). 
estimating ood accuracy using agl in diverse ensembles: 
 By constructing a diverse ensemble of foundation models, we can leverage AGL to extract precise estimates of model performances under distribution shift. We construct ensembles by collecting models trained from randomly-initialized heads (Section 3) and different base models (Section 4).For image classification, our model collection consists just linear models over CLIP representations.For text classification and question-answering, we include GPT, OPT, and Llama models individually finetuned from differently initialized heads. In Table 3 , we compare the Mean Absolute Percentage Error (MAPE) of AGL-based prediction algorithms, ALine-S and ALine-D [2] , to other baselines.We compare against confidence based methods ATC [19] , AC [24] and DOC-Feat [21] and Naive Agreement which directly uses agreement between model pairs [27, 35] . For confidence based methods, we first temperature scale the models using ID validation data, and pick the lower error rate from the estimations obtained with and without temperature scaling. However, there are several limitations when naively applying confidence baselines to estimate performance on question-answering, as they estimate classification accuracy. First, there is no easy analogous formulation of confidence baselines for the F1 score, so we estimate the exact-match score instead for fair comparison. On the other hand, AGL can predict performance across metrics accuracy, F1, and exact-match. Second, extractive question-answering is a joint classification task where models predict both the start and end token index of the answer span in the context. More details for how we calibrate baselines for this setting is provided in Appendix A.1.2.Because ALine-S and ALine-D only provide estimation guarantees where the coefficient of determination R 2 of the linear fit in agreement is strong ([2]), we filter out datasets with low R 2 ≤ 0.95. These shifts include iWildCam-WILDS, Camelyon-WILDS, and a few corruptions in CIFAR10C, CIFAR100C, and ImageNetC. We evaluate ALine-S/D for these failure cases in Appendix A.1.3. Across datasets with a high R 2 in agreement, ALine-S and ALine-D provide precise OOD performance estimates in finetuned FMs, surpassing other baselines by a large margin. This is noteworthy, especially for shifts where the agreement line is significantly off y = x, further lending to the utility of this method. Furthermore, they perform better on the question-answering task SQuAD, with the next best confidence method achieving as large as 20% higher error. 
limitations: 
 Estimating the out-of-distribution performance of foundation models has rapidly grown in importance, especially as these models are increasingly deployed in real-world use cases. Our work focuses on a promising method to enable deployers to reduce the harm of machine learning systems when they encounter OOD inputs. However, deployers should be careful to not use AGL as the only signal for OOD performance. The correlation between agreement and accuracy is not guaranteed to hold for all distribution shifts, so other metrics should additionally be used to monitor model performance. In particular for foundation models, we observe that careful choices during fine-tuning is required to observe AGL. In fact, if different pretrained checkpoints are evaluated zero-shot, without any finetuning, AGL is not able to reliably predict the performance of FMs (Appendix A.3.5). Furthermore, while we studied AGL closely for a wide array of classification/QA benchmarks, there remains other important downstream tasks such as long-form generation that we leave for future study. We also do not provide any theoretical guarantees to back our empirical findings, such as the importance of random initialization, which we leave for future work. 
conclusion: 
 We develop methods for extending AGL to foundation models to enable OOD performance prediction in this emerging paradigm. We find that utilizing AGL for performance estimation requires a careful tuning of ensemble diversity. Unlike the original paradigm of AGL, where models observed tens or hundreds of epochs of training on the ID dataset, we find that randomness in specific optimization choices, especially linear head initialization, is crucial for foundation models. In fact, in contrast to [2] , we find that linear models can also observe AGL, specifically in the CLIP representation space, suggesting that AGL may not be a neural network specific phenomena. Our conclusion on AGL also sheds light on the robustness of foundation models. First, our experiments show that light finetuning alone can corrupt models to have diverse behaviors. Next, in contrast to vision models, where previous works show different forms of pretraining lead to different slopes in the linear correlations [43] , we find that all the language models we evaluate, e.g., OPT, GPT2, GPT2-Neo, Alpaca, Llama, and Vicuna lie on the same accuracy line. This is particularly intriguing because it goes against the common wisdom that the pretraining data influences the models' "effective robustness". We leave these questions for future analysis. 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: We implement safeguards Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All models, datasets, and methods are cited. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A.1.1 AGL-Based Estimation Methods: ALine-S/DALine algorithms are the AGL-based performance estimation methods proposed in [2] . When the AGL phenomenon occurs, i.e., models observe a strong linear correlation in both ID versus OOD agreement and accuracy with matching slopes and biases, algorithms ALine-S and ALine-D effectively apply the linear transformation calculated using agreements to map the ID performances to OOD performance estimates. We describe the algorithms in more detail below.AGL Provided a collection of models F = {f 1 , f 2 , ..., f n }, AGL suggests that ID versus OOD accuracy observe a strong linear correlation if and only if ID versus OOD agreement observes a strong linear correlation and when they do, the slopes and biases match:EQUATIONΦ -1 is the probit transform used to induce a better linear fit as used in [2] and [39] . Provided access to Acc ID (f i ), Agr ID (f i , f j ), Agr OOD (f i , f j ) ∀i, j, we'd like to estimate Acc OOD (f i ) for all f i ∈ F.ALine-S The algorithm ALine-S simply estimates the the slope a and bias b of accuracy by computing the linear fit of agreement.EQUATIONWith â and b, we estimate Acc OOD(f i ) ≈ â • Acc ID (f i ) + b. This method is called Aline-S.ALine-D This method instead constructs the following system of linear equations. Provided the relation in Equation 2, one can derive that for any f i , f j ∈ F,EQUATIONTreating Acc OOD (f i ) ∀i as unknown variables, note that the right hand side is known and we can construct a linear system of equations using all n 2 pairs of models. The algorithm employs least squares to solve this approximate system of linear equations.We compare ALine against confidence-based methods ATC [19] , AC [24] , and Doc-Feat [21] . These methods notably perform better after calibrating the models ID by temperature scaling.Classification For classification tasks, we optimize a temperature T for each model f on the cross-entropy loss over the in-distribution validation data.EQUATIONwhere σ(•) is the softmax.Question-Answering For extractive question-answering tasks, the model has to predict two labelsthe start and end token index y = [y s , y e ] of the context span that answers the question. For each model, we attach a span prediction headv = [v s , v e ] ∈ R d×2 on top of the base B θ (x) ∈ R d×N where N is the token length of x. s(x) = v ⊤ s B θ (x) and e(x) = v ⊤ e B θ (x)predict the start and end token index, respectively.We're interested in evaluating question-answering models on the exact match (EM) objective,EQUATIONEM treats question-answering as a classification problem over N × N choices of start and end index pairs. This allows us to utilize confidence-based methods that are designed for classification tasks. We can calculate the model confidence for index pair [i, j] as σ(s(x)) i • σ(e(x)) j .We jointly optimize a separate temperature for the start and end logits T s and T e and we minimize the cross-entropy loss over the in-distribution validation data. In this section, we present a comparison with ProjNorm [63] , a method that yields a score which is shown to be correlated with the OOD performance of the model. We study the same setting as presented in Section 4 where we estimate OOD performance of foundation models pretrained on different text corpora. Unlike AGL, ProjNorm doesn't provide an estimate of the OOD performance hence we compare the linear correlation between the predicted value and OOD performance. From Table 4 it can be seen that estimates from ALine-D are more strongly correlated with OOD performance than ProjNorm. In our comparison with baselines in Section 5, we filter out datasets with a low correlation coefficient ≤ 0.95 in ID vs OOD agreement. When the linear correlation of agreement is weak, AGL tells us that the correlation is also low for accuracy, and AGL-based methods are not guaranteed to be reliable in such circumstances. We provide ID vs OOD accuracy and agreement scatter plots for all datasets in Appendix A.3.2.Below, we separately provide the comparison with baselines for the excluded datasets. We generally find that the baseline ATC [19] is significantly better in circumstances where AGL-based methods are unreliable. In Section 3.1, we examine how diversity source impacts whether AGL is observed in linear probed CLIP models from CIFAR10 to CIFAR10C. Here, we perform the same experiment on Office-Home [56] , which consists of 4 domains or image styles ("Art", "Clip Art", "Product", and "Real World") for 65 common objects. We train models on one domain and treat the remaining three domains as OOD. Similarly, only Random Initialization yields AGL or matching slopes in accuracy and agreement, and as a result, the corresponding MAPE of estimating the OOD performance of this diverse ensemble is the smallest. We report the strength of AGL in linear probed CLIP models with randomly initialized linear heads across all datasets discussed in Section 3.2. We show that light full finetuning over CLIP also observes similar effects of diversity source on the strength of AGL. We verify this on shifts from CIFAR10 to CIFAR10C. On the other hand, we demonstrate that in models heavily trained from scratch, AGL can be observed irrespective of the diversity source. Consistent with the models in [2] , we train ResNet18 on CIFAR10 from scratch, varying the different sources of randomness. These models are trained heavily with SGD with learning rate of 1 × 10 -2 , batch size 128, and weight decay of 1 × 10 -5 for up to 200 epochs. We do not use any data augmentation. To rid of any confounding factors from data subsetting observing a smaller amount of data, we evaluate all randomness sources on different training dataset sizes. We track the effect of diversity in random initialization, data ordering, and data subsetting for different portions of the training data (100%, 50%, 30%, 10%). For each percentage x%, Random Initialization and Data Ordering ensembles are trained on the same randomly sampled x% proportion of the data while each model in the Data Subsetting ensemble observe different randomly sampled x% portions In Section 3.2, we report the effect of diversity source in full finetuned GPT2-Medium models for the extractive QA task SQuAD to SQuAD-Shifts. In this section, we also provide the same experiments on OPT-125M and BERT. As observed in GPT, using Random Heads yields the strongest AGL behavior and achieves the smallest ALine-D MAPE.In Table 14 and 15 We track the effect of diversity in random initialization, data ordering, and data subsetting for different portions of the training data (100%, 50%, 30%, 10%). For each percentage x%, Random Initialization and Data Ordering ensembles are trained on the same randomly sampled x% proportion of the data while each model in the Data Subsetting ensemble observe different randomly sampled x% portions. In Section 3.2, we reported the effect of diversity source for full finetuned OPT. Here, we demonstrate similar results on text classification shift from MNLI-matched to SNLI with linear probed GPT2-Medium (Figure 34 ) and OPT-125M (Figure 35 ). Similarly, random initialization observes the strongest AGL behavior. In Table 16 , we report the average MAPE of OOD performance estimation using ALine across models. We also test whether AGL appears for generative question answering, where models generate the answer to the question instead of directly extracting a span from the provided context.For generative tasks, it is common to finetune over the base model directly instead of attaching a linear head on top. We again study different ways of achieving a diverse set of classifiers (e.g., Random Initialization, Data Ordering, Data Subsetting). However, we replace randomly initialization of a linear head instead with randomly initializing the non-zero initialized LoRA weights.We use a base model GPT2 and finetune models using cross-entropy loss on the next token prediction objective over SQuAD. During training, we concatenate the context, question, and answer together, and we apply the next token prediction objective on just the answer tokens. We add a line break (\n) after the answer, which functions as a "stop token" that the models also have to predict after answering the question. We train with the AdamW optimizer with a learning rate of 1e -4, weight decay of 1e -2, and batch size of 16 up to a maximum of 4 epochs. As with extractive QA, we measure the F1 score of the model's answer (its output before a line break).We track the effect of diversity for 100% of the training data in random initialization and data ordering, 50% of the training data in data subsetting. Figure 36 shows that all sources of diversity are sufficient to show AGL. Diversity is different from extractive question-answering because the search space of all tokens already provides enough diversity among models. We also note the MAE in Table 17 and MAPE in Table 18 . We also finetune base models of the GPT and OPT model families for generative question-answering using the same hyperparameters mentioned in the previous section. Similarly, we evaluate models using the F1 score. Figure 37 shows that AGL holds for all shifts in SQuAD-Shifts. Furthermore, it also holds for the mlqa-translate-test.es test split of the MLQA dataset [32] , which is the English portion of English-Spanish translated MLQA questions. AGL holds for all SQuAD-Shifts splits and MLQA Spanish split.We test the robustness of our observation across learning rate and batch size for single base FM on CLIP embeddings. Figure 38 shows that AGL holds regardless of the learning rate or batch size used for fine-tuning. That is, AGL holds with random initialization of the head and does when for shuffling the data subset or randomizing the data ordering. We compare full fine-tuning with different PEFT methods and observe that AGL holds for randomly initialized heads and does not for data ordering or data subsetting. Figure 39 shows a single GPT2 trained with LoRA [25] , IA3 [33] , and BitFit [64] . The linear fit for the three PEFT methods aligns with full fine-tuning for randomly initialized heads. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [NA] Justification: Our work has no algorithmic contributions and results can be replicated with https://github.com/kebaek/Agreement-on-the-line Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper investigates the applicability of agreement-on-the-line (AGL) to finetuned large language models for predicting out-of-distribution (OOD) performance. Reviewers generally found the paper to be well-written and technically sound, the method is simple and practical, and importantly, the empirical evaluation is extensive and done across various benchmarks and model sizes ("The authors run extensive evaluation on models of different sizes and on different downstream tasks." - 8Ly8). Reviewers raised few questions regarding the clarity of figures and explanations, or additional studies of AGL in the zero/few-shot settings and with different PEFT methods, which appears to be sufficiently addressed during the rebuttal stage. However, there is lingering concern regarding the lack of theoretical insight or sufficiently convincing explanation on the success of AGL ("Without any explanations or theoretical insights, I am doubtful whether AGL (particularly given its origin from random initialization in a convex problem) is indeed meaningful or just a mere coincidence that has trivial explanation" - H4MK), which authors may consider addressing during paper revision stage.

Reviews: review: paper_summary: paper_summary: This submission studies the problem of predicting OOD performance given known in-domain performance. Building on top of recent work showing that ensembles can be used for this problem, by looking at agreements between components in the ensemble as surrogate labels to predict OOD performance, they find that a similar approach can be used for finetuned LLMs, as long as ensembles are generated by finetuning randomly initialized heads. The topic is timely, the paper is generally clearly written (although improvements are needed) and the empirical validation is good.
strength_weakness: strength_weakness: - some critical details are missing, for instance the authors should report in the main paper how to go from Acc, Agr to OOD (line 160-164).
- the significance is unclear to me: It would be useful to predict OOD performance at larger scales as opposed for models that are finetuned for more steps.
questions: questions: If only the topmost layer is randomly initialized and trained, then all components of the ensemble should converge to the same parameter vector provided that they are trained for long enough because the optimization is a convex problem. Is the diversity due to the limited number of training step? If it is the case, I think the assumption should be made explicit.

Could the authors compare the method of sec. 4 against the method of sec 3 directly?
limitations: limitations: no concern
paper_summary: paper_summary: The authors of the paper propose a method to demonstrate that foundation models can exhibit agreement-on-the-line (AGL), under certain conditions. The existence of AGL can be used to predict the OOD capabilities of models without having access to the labels for the downstream tasks.
strength_weakness: strength_weakness: - I believe that the paper's clarity, while overall good, could be improved further in certain points:

  - Figure 1 should be a little clearer, with the caption being a little bit more detailed. This will help a lot with understanding of the key results of the paper, given the early position of the Figure in the document.

  - I feel like the authors should elaborate a bit on lines 246 - 255, given that their result here is in contrast with previous statements made in the literature. A little more discussion here would be helpful.

  - Similarly, Figure 3 should also be expanded a bit more, especially since in some settings the ID - OOD line lies directly on top of the $y= x$ axis, which I find very surprising.

- The fact that AGL can predict OOD performance, while interesting, comes with the caveat in Section 5 that it cannot be currently applied to all datasets. The authors explicitly state that and provide a criterion to determine the setting in which AGL is predictive of OOD accuracy. Nevertheless, I think this Section requires a bit more detail (see also my question below).

Overall, this is an interesting paper in my opinion, and I think my concerns with it currently are mostly based on the clarity.
questions: questions: I would like the authors to explain what the correlation $R$ in Section 5 refers to. If it is ID vs OOD agreement for various models, then it would mean that these should be nearly parallel to the $y = x$ line, which is a limitation of the setting where AGL is predictive of OOD performance.
limitations: limitations: I think the authors have adequately addressed the limitations of their work, and I cannot find any negative societal impact arising from their work.
paper_summary: paper_summary: The paper studies the applicability of agreement-on-the-line (AGL) to finetuned large models. Specifically, AGL is the phenomenon where the agreement in predictions of a collection of models on in-distribution (ID) data is linearly correlated with these models' agreement on out-of-distribution (OOD) data. This phenomenon is particularly interesting since earlier work has shown that this relation holds whenever the accuracy-on-the-line (ACL) holds; furthermore, these linear relation for agreement and accuracy is the same. While prior work has studied this phenomenon in a variety of settings, this paper studies instead this phenomenon for large finetuned models. The authors show that even in this regime, AGL holds in both vision and language settings. Furthermore, while prior work has shown that vision models pretrained on different distributions do not share the same AGL line, the authors find that the agreement of language models actually falls on the line.
strength_weakness: strength_weakness: - The paper lacks an analysis on the reason behind the different behavior observed between the vision and language models. Although these are two different modalities, I don't see a reason why vision and language models should behave differently. 
- The paper lacks an analysis of AGL in zero/few-shot settings.
- The paper assumes that full finetuning and finetuning with LoRA lead to the same AGL phenomenon. While LoRA (and other PEFT methods) lead to a similar performance, their behavior might be different [1]. Such an analysis is needed, and the inclusion of more PEFT methods (QLoRA, BitFit, IA3, etc.) would be nice.


[1] Empirical Analysis of the Strengths and Weaknesses of PEFT Techniques for LLMs. Pu et al. 2023.
questions: questions: - The authors point out that they vary several factors when finetuning the large models, including the initialization of the linear head. I am a bit confused why different initialization would lead to substantially different accuracies. For example, in figures 1 and 2 (where there's supposed to be a single backbone that's finetuned several times), we can see that the x-axis values range from 10% to 90%. Can you please explain the reason behind this wide range?
- Can you please add more results with different PEFT methods and analyze the difference (or the similarity) between them?
limitations: limitations: Yes
paper_summary: paper_summary: The paper shows that agreement-on-the-line holds in finetuned foundation models across vision and language benchmarks and finds that random head initialization is critical for the phenomena. The authors also show that agreement-on-the-line holds in ensembles of different pretrained models. They demonstrate usage of agreement-on-the-line to predict OOD performance.
strength_weakness: strength_weakness: Minor points:
1.	The panel labels in Fig. 31 and 32 has a repetition of Random Head, which should be corrected.
2.	The authors fail to discuss the robustness of the method. Specifically, it is important to know how robust the method is in terms of changing different hyperparameters.

Major points:
1.	Novelty: the authors extend a previously proposed method (Baek et al. 2022) to finetuned foundation models. Although the aspect that observing AGL holds in this new setup is to some extent is novel, the methodology itself lacks novelty. 
2.	The paper is mostly experimental results without providing any insights into why one would expect AGL to hold or correlate with ACL. Without any explanation or theoretical guarantee, it is debatable whether one can trust this method with new models or new data.
questions: questions: 1.	If you only train a linear head and freeze all the other parameters, isn’t the problem convex? Then, suppose training to convergence, initialization could potentially bias the final solution. However, this bias should be a result from the random initialization in the data kernel space. Why would this type of randomness be meaningful at all? 
2.	What do you think are potential causes for the larger slopes of AGL compared to ACL if trained with data ordering or data subsetting?
limitations: limitations: The authors have listed limitations at the end. However, I think one major limitation is when the correlation coefficient is low, other methods will outperform AGL, which is shown in Table 5. I suggest authors include this limitation.

Comments: title: Response to rebuttal
thank you
comment: Thank you very much for the detailed response to my comments! As all of the points I made have been addressed, I am raising my score a bit.
I'd like to thank the authors for their response which I find satisfactory. I hope they will revise the paper accordingly. I am still supportive of accepting this paper.
Thank you for the clarification and for running additional experiments. Adding these results to the main paper (or in the Appendix and referencing them in the main paper) would definitely strengthen the paper. I will raise my score to 7.
I thank the authors for their detailed responses! The authors have addressed my concerns around robustness with respect to the hyperparameters. Thus, I am increasing my score from 4 to 5. However, I still hold my skeptical position. Without any explanations or theoretical insights, I am doubtful whether AGL (particularly given its origin from random initialization in a convex problem) is indeed meaningful or just a mere coincidence that has trivial explanation.