Quantifying Prediction Consistency Under Fine-tuning Multiplicity in Tabular LLMs
Authors: Faisal Hamman
Pasan Dissanayake
Saumitra Mishra
Freddy Lecue
Sanghamitra Dutta
Venue: ICML 2025 poster (2025)
Keywords: Tabular LLM
Model Multiplicity
Few-shot Learning
Prediction Consistency
Primary area: social_aspects->robustness
TL;DR: We propose a local stability measure to preemptively quantify prediction consistency in Tabular LLMs under fine-tuning multiplicity, without retraining multiple models.

Abstract: Fine-tuning LLMs on tabular classification tasks can lead to the phenomenon of *fine-tuning multiplicity* where equally well-performing models make conflicting predictions on the same input. Fine-tuning multiplicity can arise due to variations in the training process, e.g., seed, weight initialization, minor changes to training data, etc., raising concerns about the reliability of Tabular LLMs in high-stakes applications such as finance, hiring, education, healthcare. Our work formalizes this unique challenge of fine-tuning multiplicity in Tabular LLMs and proposes a novel measure to quantify the consistency of individual predictions without expensive model retraining. Our measure quantifies a prediction's consistency by analyzing (sampling) the model's local behavior around that input in the embedding space. Interestingly, we show that sampling in the local neighborhood can be leveraged to provide probabilistic guarantees on prediction consistency under a broad class of fine-tuned models, i.e., inputs with sufficiently high local stability (as defined by our measure) also remain consistent across several fine-tuned models with high probability. We perform experiments on multiple real-world datasets to show that our local stability measure preemptively captures consistency under actual multiplicity across several fine-tuned models, outperforming competing measures.

Quantifying Prediction Consistency Under Tine-Tuning Multiplicity in Tabular LLMs
Faisal Hamman Pasan Dissanayake!
Saumitra Mishra Freddy Lecue Sanghamitra Dutta"
Abstract
many of these sectors rely on tabular data, Tabular LLMs (TabLLMs) is emerging research priority (van Breugel & van der Schaar, 2024). Recent studies have shown that TabLLMs perform commendably in scenarios with limited training data due to their transfer learning abilities (Hegsel- mann et al., 2023; Dinh et al., 2022; Yin et al., 2020; Yan et al., 2024; Wang et al., 2023). However, these models are often fine-tuned from large pre-trained models with millions or billions of parameters small, proprictary datasets (Hu et al., 2021; Liu et al., 2022). This paucity of training data and large parameter space introduces arbitrariness and incon- sistency across fine-tuned model variants, raising concerns about their trustworthy adoption in high-stakes applications.
Fine-tuning LLMs on tabular classification tasks can lead to the phenomenon of fine-tuning mul- riplicity where equally well-performing models make conflicting predictions on the same input. Fine-tuning multiplicity can arise due to varia- tions in the training process, e.g., seed, weight initialization, minor changes to training data, etc., raising concerns about the reliability of Tabular LLMs in high-stakes applications such as finance, hiring, education, healthcare. Our work formal- izes this unique challenge of fine-tuning multiplic- ity in Tabular LLMs and proposes novel measure to quantify the consistency of individual predic- tions without expensive model retraining. Our measure quantifies a prediction's 's consistency by analyzing (sampling) the model's local behavior around that input in the embedding space. In- terestingly. we show that sampling in the local neighborhood can be leveraged to provide proba- bilistic guarantees on prediction consistency un- der broad class of fine-tuned models, i.e., inputs with sufficiently high local stability (as defined by our measure) also remain consistent across several fine-tuned models with high probability. We perform experiments multiple real- world datasets to show that our local stability measure preemptively captures consistency under actual multiplicity across several men- models, out- performing competing measures.
One imminent challenge for TabLLMs fine-tunig multi- plicity where multiple well- sell-performin models fine-tuned from the same pre-trained LLM under slightly varying con- ditions (e.g., different seed, hyperparameters, or minor changes in training data) produce conflicting predictions for the same inputs. This concept is closely related to pre- dictive multiplicity, often referred to as the Rashomon effect in the context of decision trees and neural networks (Marx et al., 2020; Breiman, 2003; Hsu & Calmon, 2022). While multiplicity has also been observed in LLMs for text classi- fication (Gomez et al., 2024), we are particularly interested in fine-tuning multiplicity in TabLLMs (essentially minor model variations) due to their relevance in high-stakes clas- sification tasks. E.g., in areas like finance (Yin al., 2023) and healthcare (Wang et al., 2024b; Chen et al., 2023b; Kim et al., 2024), arbitrary and conflicting predictions on the same input under minor model variations can lead to confu- sion, reputational risk, and distrust among stakeholders.
Aside from the inherent need for predictions to be consistent to minor model variations (due to seed or hyperparameters), Ta deployed by institutions may also need to be up- dated for various reasons, e.g., to retrain on a few additional data points (Wu al., 2024), or even remove few data points for privacy. Regulatory frameworks like the GDPR (Voigt, 2017) introduce the right to be forgotten which allows un- learning an individual's I's data upon request, potentially lead- ing to model updates. These model updates could, in turn, impact previously issued predictions. Fine-tuning multi- plicity also paves the way for fairwashing and explanation bias (Black al., 2022: Sokol et al., 2023; Rudin al.,
1. Introduction
Large language models (LLMs) are generating significant interest in high-stakes applications, e.g., finance, healthcare, etc., particularly in few-shot classification scenarios. Since
DDpaatment of Electrical and Computer Engineering, Univer- sity of Maryland, College Park *JPMorgan Chase Al Research. Correspondence to: Faisal Hamman <fhamman @ umd.dddd... edu>>
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMS
2024), making quantifying consistency under fine-tuning multiplicity an important and practically relevant problem.
ity measure (computed preemptively without retraining or ensembling) is quite well-aligned with the consistency of data points under actual fine-tuned multiplicity for several datasets, namely, the Ge German Credit, Bank, Heart, Car, Di Diabees,, and Adult t datasets (Kahn; Hofmann, 1994; Becker & Kohavi, 1996). We employ the BIG- SCIENCE TO encoder-decoder model (Sanh et al., 2021) and Google FLAN-TS (Chung et al., 2024), fine-tuned via the T-Few recipe (Liu et al., 2022), and LoRA (Hu et al., 2021). For each case, we empirically evaluate the extent of fine-tuning multiplicity, and also study how our local sta- bility measure Sk. f), (measured only using one model f) can preemptively capture consistency under fine-tuning multiplicity better than competing measures including pre- diction confidence alone.
Existing approaches to measure multiplicity in machine learning often involve retraining and ensembling multiple models (Marx et al., 2020). However, such approaches can be computationally expensive for LLMs due to large param- eter sizes. This raises key question: Can we preemptively quantify the consistency of individual predictions under fine-tuning multiplicity without actual retraining and ensem- bling? To address this question, we propose novel measure, termed local stability, which leverages the model's ' local be- havior around each input data point in the embedding space to estimate the prediction's 's susceptibility to multiplicity. Our contributions are summarized as follows:
Study the intriguing nature of fine-tuning multiplicity in TabLLMs. We first demonstrate that prediction inconsis- tency exists when we actually fine-tune several models from the same pre-trained model, as observed through existing multiplicity measures such as Arbitrariness, Discrepancy, Pairwise Disagreement, as well as two of our proposed multiplicity measures, Prediction Variance, and Range (de- fined in Section 2). We also visualize the decision boundary for several TabLLMs fine-tuned for a simple classification task and unravel an interesting "noise" pattern: unlike neu- ral network classifiers which typically have docally-smooth decision boundaries, Tabular LLMs show abrupt and im- pulsive variations (see Fig. 2). Thus, a model having high confidence in a prediction alone does not guarantee its con- sistency under fine-tuning multiplicity.
Related Works: LLMs for tabular data is growing area of research (Yin et al., 2020; Li et al., 2020; Narayan et al., 2022; Borisov et al., 2022; Bertsimas et al., 2022; Onishi et al., 2023; Zhang et al., 2023; Wang et al., 2023; Sui et al., 2024; Yan et al., 2024; Yang et al., 2024). While neural net- works and gradient- boosted trees perform well with tabular data when ample labeled data is available, their effective- ness drops considerably in data-scarce scenarios. In contrast, LLMs can leverage their reasoning, in-context learning, and pre- tr-irraind knowledge to maintain strong performance even on tiny tabular datasets (Hegselmann et al., 2023). Dinh et al. (2022) proposes LIFT, a method for adapting LLMs to non- non-language classification and regression tasks without changing the model architecture or loss function. Hegsel- mann al. (2023) studies the use of L.Mss for zero-shot and few-shot classification of tabular data and finds that this method outperforms previous deep-learning-bassee ap- proaches and is competitive with traditional baselines like gradient- boossted trees. Wang et al. (2024b) presents Med- iTab, a method that uses LLMs to combine different medical datasets, significantly improving predictions for patient and trial outcomes. Tabular LLMs have also been applied in other high-stakes domains (Chen et al., 2023b; Kim et al., 2024; Li et al., 2023; Yin et al., 2023). Yin et al. (2023) presents FinPT, an LLM based approach to financial risk prediction. We refer to Fang et al. (2024) for a more detailed survey on LLMs for tabular data.
A measure to quantify prediction consistency under fine- tuning multiplicity. We introduce a novel measure, termed local stability (see Definition 5), to quantify the consistency of model predictions under fine-tuming multiplicity without retraining several models. Given an input x and model's prediction probability for a class c, i.e., fx(x) E [o, 11, our measure is Sk. (x, = x ExIeN,, (Ja(x.) - [f((x) fx(x:))). where Nx is set of k points sampled indepen- dently from distribution over a hypersphere of radius o centered at x. This measure uses the input's local neighbor- hood (in the embedding space) to inform the local stability, capturing both the mean model confidence in the neighbor- hood and the variability in confidence in that region.
Breiman (2003) introduced the idea that models can differ significantly while achieving similar average performance, known as the Rashomon effect. Marx et al. (2020) high- lighted the prevalence of arbitrary decisions in simple classi- fication problems, calling this predictive multiplicity. Creel & Hellman (2022) discuss the harms of predictive mul- tiplicity and arbitrary decisions. Methods such as Tree- Farms (Xin et al., 2022), CorelsEnum (Mata et al., 2022), and RashomonGB (Hsu et al.) provide tools to enumer- ate models in the Rashomon set for different hypothesis spaces. Efforts to leverage model multiplicity beneficially
Probabilistic guarantees consistency over broad class of fine-tuned models. We provide a theoretical guar- antec (see Theorem 1) that predictions with sufficiently high local stability (as defined by our measure) will remain con- sistent with high probability over a broad range of equally- well perfooeefonnnng fine-tuned models. To derive this guarantee, we make some mild assumptions on the behavior of this fine- tuned model class (see Assumption 1). Our proof leverages Hoeffding's 's Inequality (see Lemma 2).
Experimental results. We show that our Local Stabil-
2

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
(a) Finetuning LLMs for Tabular Data
Tabular data
Serialized Input
LIM
Age
Education
Gain
Incomm
Fine-tune LLM using labeled examples
24
Bacielor
2174
<30K
36
Hi-gnd
>50K
This pesson 42 years old, masters degree, and a gain of 594
Predictions
Lahels
Pretrained model
Tunable parameters
+30K
>50K
Dectorate
X50K
Master
Serialize feature names and values into natural language string
Backpropagation to update tunable parameters
(b) Finetuning Muloiplictty MunbliiingMulbplicctty Tabular LLMs
(c) Our Proposed Consistency Measure
Test Input
Embedding Layer Iredicted probabiiities
Fee
f(1z)
This person is 23 vears old, hos PhD degree, and gain
Pretrained model
Seed
Seed
No
Input
Pretrained
model
Tamble goranolirg
((z)
Sa
f(xx)
No
We samplek points in bounded neighborhood of a given input in embedding space.
Consistency measured using predictoss from perturboted points
Models fine tnn-ttune from the same pre trained LLM under sighhtlyvrryinng conditions (e.g. random seeds), exhibit comparable performance generate conflicting predictions for the same inputs.
Figure I. (a) illustrates the process of fine-tuning LLMSs for Tabular data using few labeled examples (Hegselmann et al., 2023; Dinh et al., 2022). (b) demonstrates the concept of finetuning multiplicity. Models fine-tuned from the same pre-trained LLM under slightly varying conditions, such as diffcrent random seeds, can exhibit comparable performance metries but may yield conflicting predictions for the same input. (c) introduces our proposed local stability measure designed to quantify the consistency of individual predictions without requiring the retraining of multiple models. By sampling points in bounded neighborhood around a given input in the embedding space, the consistency measure Sx (x. informs a prediction's susceptibility to multiplicity.
while addressing its implications have been explored by (Black et al., 2022; Fisher et al., 2019; Xin al., 2022: Coston et al., 2021). Model multiplicity in fairness and ex- plainability are examined by Sokol et al. (2022); Hamman et al. (2023; 2024); Black et al. (2021); Dutta et al. (2022); Pawelczyk et al. (2020). Watson-Daniels et al. (2023); Hsu & Calmon (2022) offered a framework for measuring pre- dictive multiplicity in machine learning models, however, this involves retraining several models, with the exception of (Hsu et al., 2024) who propose drop-out based approach to explore the Rashomon set for neural networks. Model multiplicity under fine-tuning has not been extensively stud- ied in TabLLMs. The closest work is Gomez et al. (2024), which empirically investigates prediction arbitrariness for text classification (online content moderation). In this work, we isolate and examine a specific form of multiplicity in TabLLMs that focuses on minor model variations due to fine-tuning from the same pre-trained LLM (see Section 2). We leverage the embedding space of LLMs to preemptively quantify consistency under fine-tuning multiplicity without expensive retraining (see Section 3). There are also other alternate directions in robustness literature that have focused on aspects other than multiplicity such as out-of-disttibution generalization, adversarial examples, and uncertainty esti- mation (Djolonga et al., 2020; Han et al., 2023).
dimensional feature vector (rows of a tabular input), and each label Yi is binary, Y4 {0,1): We focus on n-shot classification where a pre-trained model is fine-tuned on a limited number of training examples n.
Serialization of Tabular Data for LLMs: To apply LLMs to tabular data, it is crucial to transform the data into a natural text format. This process, known as serialization, involves converting the table rows into a text string that includes both the column names and their corresponding values (Yin et al., 2020; Jaitly et al., 2023; Hegselmann et al., 2023; Dinh et al., 2022). The resultant serialized string is combined with task-specific prompt to form the input for the LLM. There have been various proposed methods for serialization, and this is still topic of active research (Jaitly et al., 2023). Among the serializations we have examined are: list template (a list of column names and feature values), and text template (The <co umn name> is <va 1 <value>> ) The training process uses the natural-languaae outputs of the LLM, mapped to valid classes in the target space, as part of fine-tuning (see Fig. 1). To clarify, table values are serialized into serialize(x) and then transformed into a format understandable by the LLM, tokenize (serialize (x))) (x). which is an embedding. Since these transformations are one- to-one mappings, we denote the embedded form of input x also as X' to represent x in the embedding space. This allows us to simplify the notation and directly use to refer to the input values in the embedding space.
Preliminaries: We consider a classification task for a tab- ular dataset D) {xx, y1) 1i where each x, is a d-
3

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
2. Multiplicity in Fine-Tuune Tuned Tabular LLMs
Ds(Lo) == max Ei=n + fo(x;)))
Let f(-) = -[i()), (..))........)) : : - A denote an LLM that performs multi-class classification over C classes, where A is the C-dimensional probability sim- plex (e.g., softmax outputs). Let F denote broad class of qquall--wll-peeffrming fine-tuned models (a set of competing fine-tuned models as measured by the accu- racy), i.e, Fs- = If : err(f) I err(So) + j} where err(Jo) = N Σ i=1 ([)(xx)) # Ve] for a reference model fo (with satisfactory accuracy) and test dataset with N ex- amples. Here, f(x)= - arg maxce[c] fe(x) denotes the pre- dicted label. This is a set of fine-tuned models that perform just as well as the reference baseline classifier fo. where ò E (0, 1) is the error tolerance. The appropriate choice of 5 is application-dependdett (Marx et al., 2020).
Discrepancy measures the maximum number of predictions that could change if a reference model is replaced with a competing model. This means that, in practice, altering multiple predictions requires that all conflicting predictions come from a single competing model.
Definition 3 (Pairwise Disagreement (Black et al., 2022)).
Pairwise Disagreement assesses the variability among mod- els by measuring the proportion of instances where pairs of models within the competing set disagree: Pxs(x)= ==
| [[F Ep I[ + (x)).
Since existing measures of multiplicity focus on predicted labels, we propose two more nuanced measures that leverage the predicted probabilities of model outputs:
Fine-tuming Multiplicity. We study the nature of mul- tiplicity that arises in LLMs when fine-tuned for tabular tasks. To illustrate fine-tuning multiplicity, we conduct experiments using synthetic 2D data (see Fig. 2). While fine-tuning an LLM on such data might seem excessive, it provides clear visualization of the phenomenon. We fine-tune several competing models using the text template ( (The <column name> is <value>> and varying only the random training seed. We reveal that fine-tuning LLMs on such non-language tasks exhibit noisy and non-smooth decision boundaries, even in regions where the model is expected to confidently predict a specific class. We hypoth- esize that this noisy behavior is likely because LLMs are optimized for capturing complex language structures. When fine-tuned on tabular data tasks, which often involve both text and numeric values, LLMs leverage their pre-trained knowledge but still exhibits instabilities.
Definition 4 (Prediction Variance). PV measures the vari- ability of the model oulputs for a given inpul x and class across different models in the set Fs: PV)(x)= := [iFa] Ejer, (fc(x) 1 FFll Eye,,
Unlike threshold-basee measures, Prediction Variance cap- tures variability in predicted probabilities. We also de- fine Prediction Range to quantify the maximum spread in predicted probabilities: PR,(x)= == max fe(xx et al. (2023)).
minyers fc(x) (also see Watson-
For brevity, from now on we use fxx to refer the pre- dieted probability for the specific class of interest (typically the predicted class for x), rather than the full vector of probabilities. Thus, (x) E [0, 1] will be scalar.
3. A novel measure to preemptively capture prediction consistency
Evaluating Fine-tuning Multiplicity. To evaluate the ex- tent of multiplicity on actual fine-tuned models, we now introduce specific empirical metrics that assess how predic- tions may vary across different fine-tuned models.
Our objective is to define measure, denoted as S(x, f), for an input x and a given fine-tuned model f. that would preemptively quantify the consistency of the prediction f(x) over a broad class of equally-well-eerformnng fine-tuned models. We desire that the measure S(x, f) should be high if the predictions for the input x is consistent across this broad class of fine-tuned models (see Fig. 1).
Definition (Arbitrariness (Gomez et al., 2024)). Arbi- trariness over set F& measures the extent of confticting pre- dictions across the model space for given set of inputs {xi...., ..... It is defined as: As = 1. -.E1i IBf E Fs.: f(x,) # (xx))
Candidate Measure: Prediction confidence S(x, f) == f(x). While the prediction probability of model f()) of- fers insights into its confidence in predicting a given class, they are insufficient for assessing consistency under fine- tuning multiplicity (see Table 2. Fig. 3, i.c., data point with high f(x) or confidence can still be susceptible to multi- plicity). In our synthetic data experiments (see Fig. 2), we also observe that noisy behaviors emerge in regions where the model should be confident in its predictions, leading to conflicting outcomes across various fine-tuned models. This indicates that relying solely on an input x may not provide a reliable assessment of consistency. To address this, we
Arbitrariness generalizes the Ambiguity measure which com- putes the fraction of points where at least one model in Fs disagrees with reference model (Marx et al., 2020). Abitrariness measures the percentage of points that receive conflicting predictions from any two models within the set F3. Arbitrariness can also be defined on a single input, i.e.,
A(x,) = f' E Fo; f(x,) & (x,)).
Definition (Discrepancy). Discrepancy quantifies the maximum proportion of conflicting predictions between a reference model and any competing model in the set:
4

---

Quantifying Prediction Consistency Under FinneTTuiin Multiplicity in Tabular LLMs
Stability S
Stabiity Stx,/') Class
Original Training Data
P f=
f=
f
Glass 1
Figure 2. Decision boundaries for multiple fine-tuned models of an LLM on synthetic datasets. We fine-tuned several models by only changing the random training seed. All models achieve comparable training loss and accuracy, yet they converge to different functions, exhibiting intriguing noisy patterns (a phenomenon absent in models like neural networks which are typically locally-smooth), Interestingly, these noisy behaviors appear even in regions where the model is expected to confidently predict a specife class. Observe the location and shape of these noisy patterns vary unpredictably across the various fine- nuned models. making them a possible factor contributing to prediction multiplicity. This highlights that model predictions alone may he unreliable and motivates our perturbation-haae approach to quantify multiplicity. The last two plots illustrate the local stability measure applied to model f across classes 0 and I, i.e., f(, The local stability measure effectively highlights regions where predictions are reliable (indicated by bright yellow color) and areas where prodictions may be unstable.
propose perturbation-basee approach that leverages the local neighborhood around the input x in the embedding space, ultimately leading to our measure of local stability.
certain regions (see Fig. 2). See App. for more intuitions and properties of stability measure.
3.2. Theoretical Guarantees on Consistency
3.1. Proposed Measure: Local Stability
We present theoretical insights that motivate our proposed stability measure Sk (x, f). in quantifying the consistency of predictions across a broad class of fine-tuned models.
Definition 5 (Local Stability). For a given data point x, let f(x represent the predicted probability (e.g., softmax logits) from a model f- The local stability is defined as:
Let f(: W) represent a target function and fo(; W) denote a pre-trained model. Parameter-efficient fine-tuning meth- ods such as Low-Rank Adaptation (LoRA) adds a low-rank updates into the weight matrices of a frozen pre-trained model effectively approximate the target function (Hu et al., 2021). In this framework, the fine-tuned model is represented as F-; W + AW), where the low-rank weight updates AW are added to the frozen model. We let the fine-tuned model be a random variable, F(; W + AW), where the randomness arises from the distribution of low- rank weights AW E W. For clarity, we use capital letters (c.g., F, X,, Z) to denote random variables, while lower- case letters (e.g., x,, f, c) indicate specific realizations. For brevity, we omit the weight parametrization.
Se. (x,0 (x, = Ko
Σ X.ENN.k
[(x:)
1
Σ Nx
K
Xx (xx=(x1,X2,.....
is a set k points sampled independently from a distribution over hypersphere of radius a centered at
Our local stability measure is tied to the confidence in predicting a specific class (i.e., the probability values de- rived from softmax logits), and not on the predicted labels. It quantifies the stability of model's predictions using the local neighborhood around an input x. The first term, Ex. (xx), captures the average confidence of the model in this region. The second term, T. Ex; [xx) - f(x)), penalizes variability by measuring how much the predic- tions fluctuate within the neighborhood. By subtracting this variability from the local average confidence, the mea- sure ensures that high scores are assigned to predictions with high local neighborhood prediction confidence and low variability. This formulation is motivated by our observa- tions on synthetic data, where models exhibited irregular, non-smooth decision boundaries despite high confidence in
Assumption 1. Let f and F denote the target and adapted models, respectively. We assume:
E|F(X)JX x E[F f(x). i.e., F(x) is an
unbiased estimator of [(x).
E [x]] f(x)]l F =1 fl < a. and the expected
norm of its gradient Ex - [I (x)) F= fl < t,
where X is drawn independently from a distribution over the hypersphere B(x,a).
5

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
F and f are twice differentiable with Hessians bounded by L,, Le, [[ VV² I. I..
Conversely, large e' indicates more erratic model class. In this case, our bound becomes less informative, and the local stability measure might perform poorly for given point. We interpret our results as follows: The model class is not well-hehaved; thus, one cannot certify a small gap be- tween f(x) and our proposed measure. We do not provide guarantees for all types of model changes, as this would be challenging with only a single model. For example, if fine-tuned models do not achieve sufficient accuracy, en- counter significant variations in hyperparameter choices, or large changes in the training data, é' is likely to be large. Our focus is on multiplicity that arises due to randomness in training, such as changes in the training seed or minor adjustments in training settings (i.e., broad class of equally- well-performing fine-tuned models). In our evaluations, we do not assume any specific values for e' and consider reg- ular fine-tumed models without imposing any theoretical constraint. The proof of Theorem is provided in App. B.
Our Assumptions are motivated by recent work by (Zeng & Lee, 2023) which provides key theoretical insights into the expressive power of LoRA adaptation for transformer mod- els. Specifically, we assume that the fine-tuned model F(x) is an unbiased estimator of the target function f(x), mean- ing that across fine-tuned variations, the expectation of the model remains centered around the true function. Their re- sults also establish the expected error over bounded region, given by Ex [XX) - F = fl S a. Under certain conditions, they show the existence of adapter weights AW such that a + 0. Additionally, we assume that the expected gradient norm is bounded by I, implying that while gradi- ents may vary across fine-tuned models, they remain close to the target function in expectation over local region.
Theorem 1 (Probabilistic Guarantee). Given a data point target model f and local stability measure Sa (x, F), Under Assumption 1, and for all > é', where é' = 2(a + to) + (L.²)). We have:
4. Empirical Results
In this section, we experiment across different datasets to (i) quantify the prevalence of fine-tuning multiplicity in Tabular LLMs, and (ii) validate the effectiveness of our proposed measure in quantifying the consistency of predictions over a broad range of equally-well-peffomnnng fine-tuned models.
-ke² 32
Pr (f(x)= 2 S F) - -) 21 exp
Theoretical Guarantee Interpretation. Our stability mea- sure S(x, f') provides probabilistic guarantee that if data point x has a sufficiently high local stability score random model F, sampled from a broad class of equally- well-perforning fine-tuned models, then the prediction on the target model f(x) will be at least S(x. f) e with high probability. For example, if S(x. f) = 0.8, we can with high probability (i.e, the prediction will remain on the positive predicted side). This implies that high local stability scores are indicative of consistent predictions. The probability of the bound holding increases exponentially with the sample size k. Conversely, a low stability score does not provide significant information about the prediction's 's behavior, as it does not guarantee a lower bound on the prediction.
Datasets and Serialization. Our experiments utilize the Diabetes (Kahn), Ge rman Credit (Hofmann, 1994), Bank (Moro et al., 2014), Heart, Car, and Adult datasets (Becker & Kohavi, 1996), serialized using the Text Template, i.e., tabular entry is converted into a natural language: The <cclumn name> is <value>> This ap- proach helps align the inputs with the training distribution of LLMs, enhancing their performance in few-shot scenar- ios (Hegselmann et al., 2023; Dinh et al., 2022).
be confident that f(x) will be at least 0.8
Models and Fine-tuning Methods. We use the BIG- SCIENCE TO (Sanh et al., 2021) and Google FLAN- T5 (Chung et al., 2024) encoder-decoder models pretrained L.MMs. TO is specifically pre-trained for zero-shot generalization through multitask learning. FLAN-TS is in- struction fine-tuned on a diverse range of tasks, achieving strong performance in seo- settings. These make both models well-suited for our experiments. For fine-tuning, we adopt the T-Few recipe (Liu et al., 2022), known for its effectiveness in few-shot learning, and LoRA (Hu et al., 2021). Detailed setup can be found in App. C.3.
Goodness of Model Class. The term e' is indicative of the quality or goodness of the fine-tuned model class. A small e' indicates a well-behaved model class, suggesting that differ- ent fine-tuned models produce similar outputs in expectation within the local neighborhood of x even if predictions might vary for a given data point. Similar behavior is visualized in Figure 2, where. despite the presence of noisy variations in the decision boundaries, the local predictions around given point remain relatively consistent across models. This behavior is expected since these models are derived from the same pre-trained model and trained with the goal of achieving similar accuracy on the dataset. In this case, our local stability measure provides an informative lower bound on the predictions f(x) with a certifiably small gap.
Evaluating Extent of Fine-tuning Multiplicity. We mea- sure the extent of fine-tuning multiplicity across the various datasets and fine-tuning methods, we use the multiplicity evaluation metrics (see Section 2). To evaluate these multi- plicity metrics across our datasets, we fine-tune 40 models Tfew recipe and LoRA using different random seeds and
6

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMS
Table 1. Evaluated Multiplicity for Different Datasets and Number of Shots on BIGSCIENCE TO. Evaluated on 40 fine-tuned mod- els on T-Few recipe using different random seeds. Multiplicity observed in predictions across different fine-tuned model, even when models exhibit similar accuracy (in this setting 0.02). using LoRA achieves results in the same ballpark (sce LoRA Table 6 in App. C)
This table reports the Absolute Spearman Correlation he- tween the stability measure and various multiplicity evaluation metrics for 128 shots on the datasets. most cases, our stability measure (z, f) shows higher correlation with these multi- plicity measures compared to predicted probabilities and drop-out method, indicating that the stability measure Sk (2, f) better in- forms about the multiplicity than other measures, See full Table 8 with 64 and 512 shot cases in App. C.
Table 2.
Dataset
No.
Multiplicity Evaluation Metries (BigScIENCE TO)
Shots Arbit. Disc. Avg. Pair, Avg. Pred. Variance
Avg. Pred. Range
Avg. Model Accuracy
Dataset
Number Shols
Meusure
Arbit.
Pairwise Disag.
Prediction Variance
Prediction Range
Disag.
Adu
128
64
10%
10%
9%
7%
7%
8%
0.01
0.01
0.10
0.10
83%
84%.
Adult
128
Pred. Prob.
Drop-Out
0.67
0.74
0.62
0,83
0.30
0.69
0.54
0.81
512
11%
8%
7%
0.01
0.12
85%
Stahilitt
0.80
0.96
0.84
0.91
German
128
64
17%
18%
11%
10%
6%
6%
0.01
0.01
0.16
0.20
71%
71%
Go mman
128
Pred. Prob. 0.57
Drop-Out
0.50
0.56
0.57
0.74
0.86
0.84
0.86
512
23%
12%
7%
0.02
0.23
72%
Stability
0.54
0.54
0.87
0.87
Disbetes
128
64
29%
13%
18%
17%
10%
11%
0,03
0.04
0.13
0.31
71%
72%
D aiabeess
128 Pred. Prob. 0.88
Drop-Out
0.89
0.93
0.92
0.93
0.92
0.95
0.94
512
16%
16%
10%
0.02
0.18
78%
Stability
0.92
0.95
0.93
0.95
Bank
128
64
11%
15%
9%
8%
6%
7%
0.03
0.01
0.31
0.22
66%
75%
Bank
128
Pred. Proh.
Drop Out
0.62
0.54
0.70
0.57
0.75
0.73
0.51
0.62
512
14%
8%
7%
0.02
0.16
81%
Stability
0.79
0.84
0.87
0.86
Deart
128
64
66%
9%%
4%
4%
2%
3%
0.01
0.01
0.05
0.10
78%
83%
Beart
128
Pred. Prob.
Drop Out
0.61
0.64
0.46
0.76
0.50
0.74
0.26
0.83
512
18%
7%
5%
0.01
82%
Stability
0.89
0,90
0,97
0.87
64
19%
10%
6%
0.01
0.18
81%
128 Pred. Prob.
0.56
0.26
0.29
0.01
Car
128
16%
7%
5%
0.01
0.14
86%
Caz
Drop-Out
0.63
0.66
0.57
0.52
512
4%
2%
0.01
94%
Stability 0.97
0.91
0.93
0.94
ing the predictions of k models with different dropped-out weights). Note that these require the same number of infer- ences, hence complexity for both methods are around the same. Here are the experiments we conducted:
test on a sample set. Here are the experiments we conducted:
We evaluate multiplicity on the BIGSCIENCE TO model fine-tuned using T-Few (see Table 1).
. We evaluate multiplicity on BIGSCIENCE TO fine-tuned using LoRA (see Table 6 in App. C).
We plot the evaluated multiplicity against our stability measure, predicted probabilities, and the drop-out method. See Figure 3 for illustration on the Adult 128 shot (BIGSCiENCE TO model). For other dataset refer to Fig- ure 6, 7, 8 in App. C.
We evaluate multiplicity on FLAN-T5 model fine-tuned using F-Few (see Table in App. C).
Comparing Local Stability Measure to Evaluated Mul- tiplicity. We assess the utility of our proposed stability measure Sp (x, f) in informing the presence of fine-tuning multiplicity. This utility is measured using the Spearman correlation coefficient (see Definition 6), between our sta- bility Sy.o (x, f) (estimated on just one model) and the evaluated multiplicity (evaluated on several finetuned mod- els), e.g., Spearman S4 (x, J); (x)) across the test set. Here, our local stability measure is taken with respect to the model's predicted class for fx).
We compute the absolute spearman correlation between the stability measures and various multiplicity evaluation metrics (128-shot setting on all datasets presented in Ta- ble 2). Results on BIGSCIENCE TO model with 64 and 512 shots are presented in Table in App. C. Results for FLAN-T5 model are presented in Table in App. C.
Ablations and Hyperparameter Selection. Theorem 1 indicate that increasing the sample size k exponentially im- proves the probability that the stability guarantee holds. However, this also increases the computational cost of model inference. We use k = 30, the maximum number that fits into one inference pass on the GPU.
For comparison, we include the following base- lines: 1) Prediction probability fxx which measures the confidence of the model in predicting a given class. 2) Binary Drop-Out Method (Hsu et al., 2024): Since there are no other baselines, we adapt this Drop-out method for TabLLMs. This method drops random weights of the model to explore models in the Rashomon set (i.e., set of compet- ing models) without retraining several models. For a fair comparison, we compare our method (sampling k points in the neighborhood of our data point in the embedding space, and computing the local stability measure) to theirs (averag-
Baselines:
For the neighborhood radius a, we sampled perturbed points from a truncated Gaussian distribution with variance of 0.01, which consistently performed well across all experi- ments. To guide the choice of a, we suggest the following data-driven approach. (1) Compute Pairwise Distances: For all training samples, calculate the median distance dmed between each point and its nearest neighbors (e.g. 1: = 5)
7

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
1.00
Prediction Range
0.8
Prediction Varlance
0.100
Pairwise Disagreement
0.6
0.075
0.4
Arbitrariness
0.75
0.4
0.050
0.50
0.2-
0.2
0.025
0.25
0.0
0.000
0.0
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Stability S(x. f)
Prediction Probabilities f(x)
Drop Pred
Figure Evaluated multiplicity (assessed on 40 retrained models) versus our stability measure, predicted probabilities, and drop-out method (evaluated on one model) for the 128-shot setting on the Adult t dataset. The plots demonstrate that high local stability values correspond to low multiplicity across various multiplicity evaluation metries. Also, observe that high predicted probability values (i.e., high prediction confidence) do not imply low multiplicity. Our stability measure provides better insight into the multiplicity of predictions compared to the predicted probabilities or drop-out prediction. App. C for visualizations on other datasets.
in the embedding space. (2) Set a as fraction of dmed (e.g., = 0. 0mmm)..) This captures the natural scale of the data while ensuring perturbations stay within the local neighbor- hood. This mirrors neighborhood
Table 3. Correlations and runtimes on the Adult datasct (128-shot) (100 finetuned models with an overall training time of 456 mins). Train time refers to the total time required to train the models needed; Evaluation time includes inference and computation time of the method over the entire test set. Stability achieves high correlations with multiplicity metrics at lower computational cost.
hyperparameters used in clustering, kernel methods (Ester al., 1996; Cortes & Vapnik, 1995), and certified robustness (Cohen et al., 2019; Salman et al., 2019), which similarly rely on training data pairwise distances to set their parameters.
Measure
Arbit.
Pairwise Disag.
Pred. Var.
Pred. Range
Train
Eval.
Time
Time
We used error tolerance = 0.02, corresponding to a 2% margin accuracy deviation. Evaluating multiplicity by re- fining multiple models is computationally expensive. Thus, we limited our study to 40 models. For the drop-out rate in the baseline, we use p = 0.1 following the recommendation in Hsu et al. (2024). To evaluate the impact of varying key parameters, we conducted the following ablation studies:
Re-training
1.00
1.00
1.00
1.00
456 mins
94.7 mins
Pred. Prob.
0.61
0.39
0.63
mins
mins
Drop-Out
0.79
0.78
0.70
0.86
4.56 mins
102 mins
AWP
0.65
0.71
0.55
0.72
4.56 mins
977.6 mins
Staiility
0.81
0.96
0.80
0.9
4.56 mins
19.4 mins
f(x))² As shown in Table 10 in App. C. both alternatives yield consistently weaker correla- tions with multiplicity metries, highlighting the importance of incorporating both local mean and variability terms.
We perform an ablation study on the sample size k, ob- serving improved performance with increasing k. Detailed results are provided in Table 11 in App. C.
To evaluate the applicability of our stability measure be- yond coRA-based fine-tuning, we conduct an ablation using two alternative tuning strategies: Prompt Tuning (Lester et al., 2021) and Prefix Tuning (Li & Liang, 2021). As shown in Table 14 in App. C, our Stability measure contin- ues correlate with multiplicity, though the correlations are somewhat weaker than in the LoRA case, likely due to the limitations of these tuning methods, which are known to be less effective than LoRA in few-shot settings.
We explore the effect of varying the neighborhood radius a. Results of this ablation study are summarized in Fig. 9 and Table 12 in App. C. Best performance is observed at 1-10-2. When a is too small (10-1) we sample (almost) the same points and our local stability measure is not more informative than the prediction probability. When a is too large (10-1), ), all information about the data point lost.
We also evaluate the Drop-Out method with varying drop- out rates E {0.01, 0.1, 0.2, 0.5). The correlation values between evaluated multiplicity and the stability measures for the 512-shot setting on the D Diabetes dataset are sum- marized in Table 13 in App. C. Our stability measure out- performs the dropout method for all p values.
Computational Efficiency. We compare the computa- tional requirements of our Stability measure against, retrain- ing, dropout- Prediction probability, and Adversarial Weight Perturbation (AWP) (Hsu & Calmon, 2022) in terms of both training and evaluation runtimes. Table 3 summa- rizes the cumulative training time and the total evaluation time over the Adult test set. Figure 4 plots each method's total runtime and correlation with multiplicity metrics. We found AWP to be expensive for LLMs since each gradient- optimization step requires full forward passes on the test
To assess the contribution of the variability term in our stability measure, we compare it to two baselines that capture only local variability: (i) absolute deviation (,(x) = =E(((x) - f(x)), and (ii) squared deviation
8

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
Table 5. Breakdown of test predictions by confidence and stahility (threshold = 0.75). 41% of predictions are both confident and sta- ble, while significant 20% are confident yet unstable-revvellnn revealing cases where high confidence masks unreliability and underscoring the value of our stability measure.
9.1s corr 1.00
Re Re-training
0.40hrs
Stability
corr 0.88
1.78hrs
Drop-Out
corr 0.78
0.08hrs corr 0.56
Pred Prob.
16.37hrs
Pred. Prob.
Stability
% Test
Description
corr 0.66
AWP
High (2 0. 0.75)
High 0.75)
41%
Confident & Stable
High (2 0.75)
Low (<0.75)
20%
Conlident but Unstable
0
5
10
15
Low (< 0.75)
High 0.75)
Uncondident but Stable
Total Runtime (hrs)
Law (<c 0.75)
Low (cC0.75)
17%
Unconfident & Unstable
Figure 4. Total runtime across the Adult test dataset. Our pro- posed method (Stability) achieves significantly lower runtime com- pared to the re-training and baselines while maintaining strong average correlation with multiplicity evaluation metrics.
predictions across different fine-tuned variants, even when they exhibit similar accuracy. This multiplicity is not cap- tured by merely examining predicted probabilities, as pre- dictions with high confidence can still be susceptible to multiplicity (see Fig. 3). Our stability measure, Sk.o (x, f),, was compared with the prediction probabilities f(x). The results, presented in Table 2,8.9, demonstrate that our sta- bility measure consistently shows mainly higher correlation with multiplicity metrics across all models and datasets compared to prediction probabilities and drop-out method. This indicates that o f) is more informative than the baselines in informing the fine-tuning multiplicity. The drop-out method is however better than the prediction prob- abilities alone. We hypothesize that our method is more suitable for LLMs because the embedding space of LLMs is significantly smaller than the parameter space (possibly more informative also). The drop-out method might need significantly more inferences to compete due to this.
Table Mean and std stability values for correctly vs. incor- rectly classified data points on the Hospi tt 1 dataset. Stability achieves a larger separation between correct (0.8710) and incor- rect (0.5729) data points than baselines, suggesting it is better at discriminating against unreliable predictions.
Method
Correct
Incorrect
Mcan
Std
Mcan
Std
Stahility
0.8710
0.1465
0,5729
0,1458
Pred. Prob
0.8994
0,2160
0.7965
0,2256
Drop-Out
0.8190
0,2832
0.7217
0.1929
set to enforce Rashomon-set constraints, incurring heavy costs. Dropout requires
inference and a prior check to ensure all dropped-out models (the models to be aggregated) are in the competing model set, hence our method would be more computationally efficient under the same kt Stability achieves the best corriae-oorrelttoo trade-off compared to baselines (see Fig. 4).
We study the unique nature of fine-tuning multiplicity in Tabular LLMSs. Marx et al. (2020); Rudin et al. (2024) argue for the necessity of measuring and reporting multiplicity to better inform predictions. Traditional methods measure multiplicity in classical ML are impractical for LLMs due to the computational challenge of retraining several fine-tuned models (Marx et al., 2020; Hsu & Calmon, 2022; Watson- Daniels et al., 2023). Our proposed measure, which requires only the given model and leverages the embedding space to inform multiplicity, addresses this issue. This approach reduces the complexity from retraining and inference to just inference, making it more feasible to apply in practice. Al- though, a large k (number of sampled points) may be needed for accurate stability estimation, it remains computationally more efficient than retraining multiple models. Compared to existing methods, our stability measure achieves a superior trade-off between runtime and correlation with evaluated multiplicity metrics, as shown in Figure 4. Our work pro- vides practitioners with meaningful information about the multiplicity of predictions, which may lead them to carefully evaluate which predictions to trust and which to treat with caution. Our research has significant implications in several high-stakes applications, e.g., hiring, finance, education, etc., where inconsistent predictions can lead to distrust.
Scalability to Large Datasets. We evaluate our method on the Hospital Readmission dataset (Gardner et al., 2023)(~100k data points, 50+ features). Retraining 40 mod- els to evaluate multiplicity would takes over 5 days (3.5 hrs/model), while our method requires only a single model and fast forward-pass sampling step. We train one model and compare Stability and baselines across correctly and incorrectly classified test points (see Table 4). Stability achieves a larger separation between correct and incorrect data points than baselines. We can also use our measure to analyze data points that are both confident and stable, or identify those that appear confident but are actually unstable. In Table 5, we grouped predictions based on their confi- dence and stability values. Observe that while 41% % of the predictions were both confident and stable, notable 20% of the predictions were confident but unstable, indicating that high confidence alone is not enough.
Discussion. Our multiplicity evaluation metrics, summa- rized in Table 1,6,7, reveal significant variability in model

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMS
Acknowledgements
updates in a dynamic environments where data constantly evolves (Amba Hombaiah al., 2021; Wu et al., 2024; Wang et al., 2024a). Lastly, our work could play role in mitigating fairwashing risks and explanation bias (Black et al., 2022; Sokol et al., 2023; Rudin et al., 2024). This transparency is crucial for maintaining ethical standards and trustworthiness Al deployment (Chamola et al., 2023).
This paper was prepared for informational purposes in part by the CDAO group of JPMorgan Chase & Co and its affil- iates ('J.P. Morgan") and not a product of the Research Department of J.P. Morgan. J.P. Morgan makes represen- tation and warranty whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the infor- mation contained herein. This document is not intended as investment research or investment advice, or a recom- mendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or ser- vice, or to be used in any way for evaluating the merits of participating in any transaction, and shall not constitute solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or to such person would be unlawful.
Limitations. Our work provides measure to assess fine- tuning multiplicity, but does not directly resolve this issue. Future research could focus on mitigation methods to ensure more consistent model predictions. A key constraint is the applicability to higher-dimensional datasets due to the lim- ited context window size of current LLMs, though extending context windows is an active area of research (Peng et al., 2023; Chen et al., 2023a). Additionally, our method's perfor- mance can be sensitive to hyperparameters, such as sample size and neighborhood radius; incorrect choices may lead to an inaccurate assessment of robustness. Our approach also assumes access to the embedding space, limiting its application to open-source models. Furthermore, the bound in Theorem is not directly computable. Estimating these unknowns such e' could be a direction for future work. Despite these limitations, our measure serves as a crucial step toward understanding and quantifying fine-tuning mul- tiplicity, laying the groundwork for future advancements.
Impact Statement
Broader Societal Impacts. The application of LLMs to tab- ular data, particularly in high-stakes domains such as finance and healthcare. presents both opportunities and risks (Bom- masani et al., 2021). Our work aims to address one of the critical challenges associated with these models: the insta- bility introduced when fine-tuning large models on small datasets. This instability, manifested as overfitting and mul- tiplicity, can undermine the reliability of model predictions in scenarios where stability is crucial. By measuring multi- plicity, our work contributes to the responsible deployment of L.MMs in domains where erroneous predictions can have severe consequences (Bommasani et al., 2021; Creel & Hell- man, 2022). Tabular data is central to these high-stakes domains but remains underexplored compared text and vision (Hegselmann et al., 2023). Recent work emphasizes the need for reliable foundation models in this modality (van Breugel & van der Schaar, 2024).
References
Amba Hombaiah, S., Chen, T., Zhang, M., Bendersky, M., and Najork, M. Dynamic language models for continu- ously evolving content. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining, pp. 2511-25224 2021.
Becker, B. and Kohavi, R. Adult. chine Learning Repository, 1996.
UCI Ma-
DOI:
http://bo..og///22333///XX2
Our approach also supports regulatory compliance by enhancing transparency and accountability in automated decision-making systems. Quantifying prediction consis- tency aligns with regulations such as the General Data Pro- tection Regulation (GDPR) (Voigt, 2017) and upcoming Al legislation, which increasingly demand explainable and reliable AI models (Chamola et al., 2023). While LLMs are more computationally expensive than traditional mod- cls, our method reduces the costs of assessing multiplicity. By avoiding repeated retraining, it enhances cost efficiency and minimizes environmental impact, lowering both energy consumption and carbon footprint (Luccioni al., 2023).
Bentkus, V. On hoeffding's inequalities. Annals of proba- bility, pp. 1650-1673, 2004.
Bertsimas, D., Carballo, K. Ma, Y., Na, L., Boussioux, L., Zeng, C., Soenksen, L. R., and Fuentes, I. Tabtext: a systematic approach to aggregate knowledge across tabular data structures. arXiv preprint 2rX1v:2206.10381. 2022.
Black, E.,, Wang, Z., Fredrikson, M., and Datta, A. Con- sistent counterfactuals for deep models. arXiv preprint arXiv:21 2021.
Furthermore, observing the nature of fine-tuning multiplic- ity in Tabular LLMs pave the way for future research into model stability. It also facilitates continual learning by in- forming the robustness of a prediction to potential model
Black, E., Raghavan, M., and Barocas, S. Model multiplic- ity: Opportunities, concerns, and solutions. In Proceed- ings of the 2022 ACM Conference on Fairness. Account- ability, and Transparency, pp. 850-863, 2022.
10

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg. J., Bosse- lut, A., Brunskill, E., et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07256. 2021.
Detrano, R., Janosi, A., Steinbrunn, W., Pfisterer, M., Schmid, J.-J., Sandhu, S., Guppy, K. H., Lee, S., and Froelicher, V. International application of new proba- bility algorithm for the diagnosis of coronary artery dis- ease. The American jouraal cardiology. 64(5)3304-300 1989.
Borisov, V., Seßler, K., Leemann, T., Pawelezyk, M., and Kasneci, G. Language models are realistic tabular data generators. arXiv preprint arXiv:2210.062 06280, 2022.
Dinh, T., Zeng, Y., Zhang, R., Lin, Z., Gira, M., Rajput, S., Sohn, J.-y,, Papailiopoulos, D., and Lee, K. Lift: Language-interfaced fine-tuning for non-language ma- chine learning tasks. Advances in Neural Information Processing Systems, 35:11763-11788 2022.
Breiman, L. Statistical modeling: The two cultures. Quality control and applied statistics, 48(1):81-88) 2003.
Chamola, V., Hassija, V., Sulthana, A. R., Ghosh, D., Dhin- gra, D., and Sikdar, B. A review of trustworthy and explainable artificial intelligence (xai). IEEE Access, 2023.
Djolonga, J., Hubis, F., Minderer, M., Nado, Z., Nixon, J., Romijnders, R., Tran, D., and Lucic, M. Robust- ness Metrics, 2020. URL ht //gtt/b..//ttttt/// com/ google research
Chen, S., Wong, S., Chen, L.,, and Tian, Y. Extending context window of large language models via positional interpolation. arXiv preprint arXiv:2306. 15595, 2023a.
Dutta, S., Long, J., Mishra, S., Tilli, C., and Magazzeni, D. Robust counterfactual explanations for tree-based ensem- bles. In International Conference on Machine Learning, pp. 5742-5756. PMLR, 2022.
Chen, Z., Balan, M. M., and Brown, K. Language models are few-shot learners for prognostic prediction. arXiv preprint arXiv:2302. 22692, 2023b.
Ester, M., Kriegel, H.-P., Sander, J., and Xu, X. A density- based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second Inter- national Conference on Knowledge Discovery and Data Mining, KDD'96, pp. 226-231. AAAI Press, 1996.
Chung. H. W., Hou, L., Longpre. S., Zoph. B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction-finttune language models. Journal of Machine Learning Research, 25(70):1 2024.
Fang, X., Xu, W., Tan, F. A., Zhang, J.. Hu, Z., Qi. Y., Nick- leach, S., Socolinsky, D., Sengamedu, S., and Faloutsos, C. Large language models (Ilms) tabular data: Predic- tion, generation, and understanding-a survey, arXiv preprini arXiv:2402. 1744,, 2024.
Cohen, J., Rosenfeld, E., and Kolter, Z. Certified ad- versarial robustness via randomized smoothing. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceed- ings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learn- ing Research, pp. 1310-1320. PMLR, 09-15 Jun 2019. URL https: / http///ooceeenggseeeee mlr press/v97/ cohen1 9c. html.
Fisher, A., Rudin, C., and Dominici, F. All models are wrong, but many are useful: Learning a variable's 's impor- tance by studying an entire class of prediction models simultaneously. Journal of Machine Learning Research, 20(177):1-88 2019.
Cortes, C. and Vapnik, V. Support-vector networks. Mach. Learn., 20(3):273-997 September 1995. ISSN 0885-
Gardner, J., Popovic, Z., and Schmidt, L. Benchmarking distribution shift in tabular data with tableshift. Advances in Neural Informaiinn Processing Systems, 2023.
6125. doi: 10. 10.1023/A:102222441444 URL https: ///oi..or///e
Gomez, J. F., Machado, C. V., Paes, M., and Calmon, F. P. Algorithmic arbitrariness in content moderation. arXiv preprint arXiv:2402. 16979, 2024.
Coston, A., Rambachan, A., and Chouldechova, A. Char- acterizing fairness over the set of good models under selective labels. In Meila, M. and Zhang, T. (eds.), Pro- ceedings of the 38th International Conference Ma- chine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 2144-2155. PMLR, 18-24 Jul 2021. URL. pwo./eooceeeeeeooooooooooooeee.c ngs. www.mmrr..... wall..press. press/ v139/ 0239/costoon2a..11
Hamman, F., Noorani, E., Mishra, S., Magazzeni, D., and Dutta, S. Robust counterfactual explanations for neural networks with probabilistic guarantees. In International Conference on Machine Learning, pp. 12351-12367. PMLR, 2023.
Creel, K. and Hellman, D. The algorithmic leviathan: Arbitrariness, fairness, and opportunity in algorithmic decision-making systems. Canadian Journal of Philoso- phy, 52(():26 43, 2022.
Hamman, F., Noorani, E., Mishra, S., Magazzeni, D., and Dutta, S. Robust algorithmic recourse under model mul- tiplicity with probabilistic guarantees. IEEE Journal on Selected Areas in Information Theory, 2024.

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
Han, T., Srinivas, S., and Lakkaraju, H. Efficient estimation of the local robustness of machine learning models. arXiv preprint arXiv:2307.13885 2023.
pp. 3045-3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Computa- tional Linguistics. doi: 10.18633/.12022.cmmmppmmm 243. URL https: emn iaaaii..24. h43//
Hegselmann, S., Buendia, A., Lang. H., Agrawal, M., Jiang, X., and Sontag, D. Tabllm: Few-shot classification of tabular data with large language models. In International Conference Artificial Intelligence and Statistics, pp. 5549-5581, PMLR, 2023.
Li, X., Chen, B., Hou, L., and Tang, R. Ctrl: Connect tabular and language model for ctr prediction. arXiv preprint arXiv:2306. 0284066002111.... 2023.
Hofmann, H. Statlog (German Credit Data). Machine Learning Repository, 1994. DOI: https///bi.ogg//.24////CCC
UCI
Li, X. L. and Liang, P. Prefix-tuning: Optimizing continu- ous prompts for generation. Zong, C., Xia, F., Li, W., and Navigli, R. (eds.), Proceedings of the 59th Annual Meeting the Association for Computational Linguistics and the 11th International Joint Conference Natu- rul Lunguage Processing (Volume I: Long Pupers), pp. 4582-4597, Online, August 2021. Association for Com- putational Linguistics. doi: 10. 100118663/1112221acl 353. URL a cel-long 353/.
Hsu, and Calmon, F. Rashomon capacity: metric for predictive multiplicity classification. Advances in Neural Information Processing Systems, 35:28988- 29000, 2022.
Hsu, H., Brugere, I. Sharma, S., Lecue, F., and Chen, C.-F. Rashomongh: Analyzing the rashomon effect and miti- gating predictive multiplicity in gradient boosting. In The eiirty-eighth Annual Conference Neural Information Processing Systems.
Li, Y., Li, J., Suhara, Y., Doan, A., and Tan, W.-C. Deep entity matching with pre-trained language models. arXiv preprint arXiv:2004.00584, 2020.
Hsu, H., Li, G., Hu, S., and Chen, C.-F. Dropout-base rashomon set exploration for efficient predictive multi- plicity estimation. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: / //openrevien ew net/ forum? F=SF222PUXX0
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. A. Few-shot parameter-efficient fine- tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems, 35: 1950-1965, 2022.
Hu, J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, I.., and Chen, W. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 06665, 2021.
Luccioni, A. S., Viguier, S., and Ligozat, A.-L. Estimating the carbon footprint of bloom, a 176b parameter language model. Journal of Machine Learning Research, 24(253); 1-15, 2023.
Jaitly, S., Shah, T., Shugani, A., and Grewal, R. S. Towards better serialization of tabular data for few-shot classifica- tion. arXiv preprint 22
Marx, C., Calmon, F., and Ustun, Predictive multiplicity in classification. In International Conference on Machine Learning, pp. 6765-6774, PMLR, 2020.
2023.
Kadra, A., Lindauer, M., Hutter, F., and Grabocka, J. Well- tuned simple nets excel on tabular datasets. Advances in neural information processing systems, 34:23928-23944 2021.
Mata, K., Kanamori, K., and Arimura, H. Computing the collection of good models for rule lists. arXiv preprint arXiv:2204.1 12285.11222..... 2022.
Kahn, M. Diabetes. UCI Machine Learning Repository. DOL: https//ddo..rg/11.244522
Moro, S., Cortez, P., and Rita, P. A data-driven approach to predict the success of bank telemarketing. Decision Support Systems, 62:22-31, 2014.
Kim, Y., Xu, X., McDuff, D., Breazeal, C., and Park, H. W. Health-llm: Large language models for health prediction via wearable sensor data. arXiv preprint arXiv:2401.066 06866. 2024.
Nalbandyan, G., Shahbazyan, R., and Bakhturina, E. Score: Systematic consistency and robustness evaluation for large language models. arXiv preprint arXiv:2503.00137, 2025.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for parfmeter-efficient prompt tuning. In Moens, M.-F,, Huang, X., Specia, L., and Yih, S. W.-t. (eds.), Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing,
Narayan, A., Chami, I., Orr, L., Arora, S., and Ré, C. Can foundation models wrangle your data? arXiv preprint arXiv:2205.099 201220000999... 2022.
12

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
Novikova, J., Anderson, C., Blili-Hamelin, B., and Majum- dar, S. Consistency in language models: Current land- scape, challenges, and future directions. arXiv preprint arXiv:2505.00266 2025.
Voigt. The eu general data protection regulation (gdpr). Practical Guide, 1st Ed., Cham: Springer International Publishing, 1031152676) 10-5555, 2017.
Wang. L.., Zhang, X., Su, H., and Zhu, J. A comprehen- sive survey of continual learning: theory, method and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024a.
Onishi, S., Oono, K., and Hayashi, K. Tabret: Pre-training transformer-based tabular models for unseen columns. arXiv preprint wrXiv:2303.15743 2023.
Pawelczyk, M., Broelemann, K., and Kasneci, G. On coun-
Wang, R., Wang, Z., and Sun, Unipredict: Large language models are universal tabular predictors. arXiv preprint arXiv:2310.0322 2023.
terfactual explanations under predictive multiplicity. In Conference on Uncertainty in Artificial Intelligence, pp. 809-818 PMLR, 2020.
Wang, Z., Gao, C., Xiao, C., and Sun, J. Meditab: Scaling medical tabular data predictors via data consolidation, enrichment, and refinement, 2024b.
Peng. B., Quesnelle, J., Fan, H., and Shippole, E. Yarn: Efficient context window extension of large language models. arXiv preprint arXiv:2309.0007 2023.
Wattonn-aaiccls J., Parkes, D. C., and Ustun, B. Predictive multiplicity in probabilistic classification, 2023.
Raj. H., Rosati, D.. and Majumdar, S. Measuring reliability of large language models through semantic consistency. arXiv preprint X21::2211.008555 2022.
Wu, T., Luo, L., Li, Y.-F., Pan, S., Vu, T.-T., and Haffari, G. Continual learning for large language models: A survey, 2024.
Raj. H., Gupta, V., Rosati, D., and Majumdar, S. Improving consistency in large language models through chain of guidance. arXiv preprint arXiv:2502. 15924, 2025.
Xin, R., Zhong, C., Chen, Z., Takagi, T. Seltzer, M., and Rudin, C. Exploring the whole rashomon set of sparse decision trees. Advances in neural information processing systems, 35:1407| 1-10044, 2022.
Rudin, C., Zhong, C., Semenova, L., Seltzer, M., Parr, R., Liu, J., Katta, S., Donnelly, J., Chen, H., and Boner, Z. Amazing things come from having many good models. arXiv preprint arXiv:2407.0483 2024.
Yan, Zheng, B., Xu, H., Zhu, Y., Chen, D., Sun, J., Wu, J., and Chen, J. Making pre-trained language models great on tabular prediction. arXiv preprint arXiv:2403.014 2024.
Salman, H., Li, J., Razenshteyn, I., Zhang, P., Zhang, H., Bubeck, S., and Yang, G. Provably robust deep learning via udversarially trained smoothed classifiers. Advances in neural information processing systems, 32, 2019.
Yang, Y. Wang, Y. Sen, S., Li, L., and Liu, Q. Un- leashing the potential of large language models for pre- dictive tabular tasks in data science. arXiv preprint arXiv:2403.202000 2024.
Sanh, V., Webson, A., Raffel, C., Bach, H.. Sutawika, L., Alyafeai, Z., Chaffin, A., Stiegler, A., Scao, T. L., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.082077 2021.
Yin, P., Neubig, G., Yih, W.-t,, and Riedel, S. Tabert: Pre- training for joint understanding of textual and tabular data. arXiv preprint arXiv:2005.083: 08314, 2020.
Sokol, K., Kull, M., Chan, J., and Salim, F. D. Fairness and ethics under model multiplicity in machine learning. arXiv preprint arXiv:2203.07139 2022.
Yin, Y., Yang, Y., Yang, J., and Liu, Q. Finpt: Financial risk prediction with profile tuning on pretrained foundation models. arXiv preprint arX1v:2308.000055 2023.
Sokol, K., Kull, M., Chan, J., and Salim, F. D. Cross-model fairness: Empirical study of fairness and ethics under model multiplicity, 2023.
Zeng, Y. and Lee, K. The expressive power of low-rank adaptation. arXiv preprint arXiv:2310.17513 2023.
Sui, Y., Zhou, M., Zhou, M., Han, S., and Zhang, D. Table meets Ilm: Can large language models understand struc- tured table data? a benchmark and empirical study. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pp. 6554, 2024.
Zhang, H., Wen, X., Zheng, S., Xu, W., and Bian, J. Towards foundation models for learning on tabular data. arXiv preprint arXiv:2310.07338, 2023.
van Breugel, B. and van der Schaar, M. Why tabular founda- tion models should be a research priority. arXiv preprint arXiv:2405.01 147, 2024.
13

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
Model Decision Boundary
Deta (y=O) Deta (y-1)
Data (ymo) Data (y-1)
Model Decision Bountery
point high output value but prone to mullppicily
Data point a highly vartable region
(a) Data point is close to the boundary.
(b) Data point lies in a highly variable region.
Figure 5. Additional motivation for local stability measure. Our measure relics both local variability and mcan confidence as they capture synergistic aspects of prediction robustness.
A. Additional Intuition Behind the Stability Measure
How stability differs from existing robustness measures: Our focus model multiplicity distinguishes this work from traditional robustness measures, which address different aspects of model behavior such as out-of-distributio (OOD) generalization, stability under natural perturbations, and uncertainty estimation (Djolonga et al., 2020). OOD generalization typically evaluates how well a model performs on data that differs from the training distribution (e.g., classifying objects seen from novel viewpoints or in cluttered settings). This often quantified using test datasets with altered conditions or domain shifts, and methods like domain adaptation are employed to enhance robustness. Stability under natural perturbations assesses the sensitivity of predictions and predicted probabilities small, random changes in the input, such Gaussian noise image transformations. Uncertainty estimation, the other hand, focuses calibrating the predicted probabilities to reflect true likelihoods, often using measures like Expected Calibration Error or entropy- -based metries to evaluate how well the model quantifies confidence in its predictions. While these methods provide valuable insights into different facets of robustness, their goals differ significantly from ours.
Han et (2023) is more closely related to our approach, as quantifies robustness by measuring the fraction of stable predictions within local neighborhood. While both approaches leverage the neighborhood around data point, the objectives diverge: Han al. (2023) focuses on quantifying the probability of stable predictions against perturbations evaluate robustness noise. In contrast, our measure aims to capture the consistency of predictions (multiplicity) among competing models within the Rashomon set. Additionally, our stability measure's 's unique mean-variance nature further distinguishes it (see Figure 4), Unlike existing metrics, it not only accounts for the average prediction within neighborhood but also penalizes the variability in predictions. Moreover, we provide theoretical guarantees on the consistency of predictions with high stability scores over a broad range of equally-well performing models. Recent work has also explored consistency LLMs across repeated inference runs or under slight semantic perturbations the input (Novikova et al., 2025; Raj et al., 2025; 2022; Nalbandyan al., 2025).
B. Proof of Theoretical Guarantee
Theorem 1 (Probabilistic Guarantee). Given a data point x, a target model J and local stability measure Sx (x, F). Under Assumption 1, and for all > é', where é' 2(x + to) ) + (Lo²). We have:
kcc
Pr (f(x) 2 (x, 21 exp
(I)
32
Proof. To prove Theorem 1, we begin with Lemma 1.
Assume the tne-ttned models F' belong to a discrete class of random variables. A specific model realization is represented as f for 1,2..... (Fl. with the complete set denoted by F f2 f2...., Fll. Each model f' is selected with
probability p.. where E pt = 1.
Lemma 1. Given Z, FXX)) -f(x)) 7(x)) F(x) and Z D Zú, under Assumption 1,
14

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
for any & > e' have:
k(ē c')
Pr(Z 2 2 exp
(2)
32
Lemma (Hoeffding's Inequality). For given random variable X; such that X; E [a, b] almost surely, and for any > 0,
k x B(X)) > >e) S <2exp (-10 (T6--a) a)2
2k
Pr
(3)
See Bentkus (2004) for detailed proof of Hoeffding's Inequality.
Since F.) E [0, 11, we have Z; E [- Hence, from Lemma 2, we have:
ke
Pr
fll F f) = exp
(4)
8
Given | [ = f]1 < we have - R fl é V (see Lemma 3). Now observe that:
- kē²
Pr(Z 2 é' + 2| f) Pr(Z B exp
(5)
Here, (a) holds since E F = /1 < é' The event on the left a subset of that on the right. Therefore, the probability of
the event {Z 2 é' + 2) occurring cannot be more than the probability of the event {2 2 E F F fl 2) occurring.
Pr(Z 2 +
Pr(Z 2 2 f
P((==
(6)
Σ Pr(F= 1*)
exp
8
kE
exp
(8)
hi( e7)2
exp
(9)
32
Here, (b) holds from the law of total probability. Next, (c) follows from (5). Finally, (d) holds from using the inequality
422 > (E + E') which holds for ē > e' > O. Setting e = ē E
have:
k Pr T Σ f(X.) 2 Σ
(F(X.) - ( - + + - f(x)) -e) -e) 2 11 - exp
(-ke²
(32
(10)
32
1=1
Observe that f(x) 2 f(x:) 1f(xx) f(xx). This applies directly from the reverse triangle inequality, i.c.. for any real
numbers and b, we have:/a| |b| la b). Hence,
f(x) (F(.)) 15(x)) - F(x)))
(1I)
Therefore, plugging (11) into (10), we have:
1 $
Pr > - F(x) +10++15((3) f(x) - -I((,) - f(x) )))
(12)
K 1=1
-ke?
=Pr(f(x) >
Σ -F(())-IFCC F(x)) - -)) 1-exp
(13)
32
15

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
Given Sk (x, F) (F(X)) IP(x) F(X,)); ), have:
-ke²
Pr (f(x) 2 (x F) -) 2 exp
(14)
32
Using Lemma 3, we show that E|Z|F- - fl S e' which completes the proof.
Lemma 3. Let F. f be twice differentiable functions with Hessians bounded by L, i.e., [V2 [Vxx) S for all
Let Ex, (X),)) F( = f] v and Ex, V (F - (xx = f] - 1, where X; drawn uniformly from
distribution over the hypersphere (3 If [(X.) - f(x.)) - + FXX.) F(x)], then,
E B F f] S to O Hessian
(15)
Proof. We first bound [XF((X)) f(X.]]: Since ( - -(x),) < a for all it follows that ([F(X,) - [(X)) Sa
Next, we bound (x,, - F(x)]
(f(,)) f(x]]:
Expand F(X;) and f(X;) around x using Taylor's 's expansion:
F(x) F VxXX)" x,) 1 _xx xx- x X,),
(16)
f(x) f(X,) (x- X,)
(x-
2 (x- X.) \ X.).
some Er E B(x, o). The absolute differences become:
1 [F(X)) - F(x) s VF(X.) Xi)) +
2 _ x||
(17)
If(X,) [(x) V](X) X.) - - L1X; - x112.
Subtracting these:
[F(X)) - F(x) If(X,) f(x)) s (x- -f(x)) + - x112.
(18)
Using the reverse triangle inequality |al |b| s la bl:
FF(X)) - F(x) - f(x) = VF (xx,) xx)) + +I|X, - -x122
(19)
Taking Expectation:
E (F -XX,) X,)1 - -XXIIXX -x11 (F S to
(20)
Hessian Term: | - x|| L- -((O))
Combining the inequalities, we have:
F = fl a O(Lo))
(21)
16

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
C. Expanded Experimental Section
C.1. Relevant Definition
Definition 6 (Spearman Correlation). Spearman's 's correlation, Spearman( (, Y), measures the strength and direction of monotonic relationship between two variables. is the Pearson correlation coefficient of their ranked values. Given n pairs (X Y)), it is computed as:
Spearman(X,, (X,)) = 6) df nnn²
cov(rank (X), (Y))
raak(XX rank(Y)
where d; is the difference between the ranks of X; and Y, The value ranges from -1 (perfect negative monotonicity) to (perfect positive monotonicity), with indicating monotonic relationship.
C.2. Dataset Details
This section provides detailed descriptions of the datasets used in experiments.
Adult dataset (Becker & Kohavi, 1996), also known as the "Census Income" dataset, is used for predicting whether an individual earns more than $50,000 annually based on various demographic attributes. It consists of 48,842 instances with 14 attributes, including age, work class, education, marital status, occupation, relationship. race, sex, capital gain, capital loss, hours per week, and native country. The dataset is commonly used in classification tasks.
German Credit dataset (Hofmann, 1994) is used for credit risk evaluation. It consists of 000 instances with 20 attributes, which include personal information, credit history, and loan attributes. The target variable indicates whether the credit is good bad. This dataset is often used for binary classification problems and helps in understanding the factors affecting creditworthiness. The dataset is commonly used in classification tasks.
Diabetes dataset (Kahn) is used for predicting the onset of diabetes based on diagnostic measurements. It contains 768 instances with 8 attributes, including the number of pregnancies, glucose concentration, blood pressure, skin thickness, insulin level, body mass index (BMI), diabetes pedigree function, and age. The target variable indicates whether the individual has diabetes. The dataset is commonly used in classification tasks.
Bank dataset (Moro et al., 2014) is used for predicting whether client will subscribe to term deposit based data from direct marketing campaigns of a Portuguese bank. It includes 45,211 instances in the training set and 18 attributes, such as age, job type. marital status, education, credit balance, housing loan status, and contact details from the marketing campaigns. The target variable indicates whether the client subscribed to the term deposit. This dataset is commonly used in binary classification tasks.
Heart dataset (Detrano et al., 1989) contains data from four different hospitals. includes 918 patients, each represented by 11 clinical variables, with the task being binary classification of coronary artery disease. Among the patients, 508 labeled positive for the condition.
Car dataset (Kadra et al., 2021) contains entries describing various cars characterized by six attributes. The task is a classification problem aimed at evaluating the state of each car. The dataset comprises 1,728 examples.
Hospital Remission dataset (Gardner et al., 2023) contains 99,493 inpatient encounters from 130 U.S. hospitals (1999-2008) that involve diabetic patients. The dataset provides 50+ attributes aimed predicting whether a patient will be readmitted within 30 days after discharge. is commonly used for binary classification tasks assessing post-discharge risk and care quality.
C.3. Experimental Setup
Our experiments were carried out using the BIGSCIENCE TO and Google Flan T5 models fine-tuned on several datasets. The number of shots was set to 64,128, and 512 for each dataset. To evaluate multiplicity and local stability, we fine-tuned 40 models with different random seeds for each dataset and recorded their predictions. The training process involved setting the batch size to 2 for smaller training sizes and 8 for larger sizes. The learning rate was set to 0.003. For each dataset, we determined the number of training steps adaptively based on the number of shots, ensuring sufficient iterations for model convergence. Specifically, the training steps were calculated as 20 x (number of shots/ batch size). All experiments were performed on 2 NVIDIA RTX A4500 and NVIDIA RTX 6000 GPUs. To ensure reproducibility and robustness of the
17

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
results, different random seeds (i.e., 2, 4, 8, etc) used for each fine-tuning iteration. For fine-tuning with LoRA rank of 4. Given the infeasibility of computing the exact size of [Fa due its potentially model space, employ expensive sampling approach, i.e., fine-tuning with various seeds. We select finite number of models from Fs for practical evaluation, allowing us to evaluate the multiplicity metrics. It is very computationally expensive to fine-tune several models to evaluate multiplicity. This motivates the need for measure to quantify stability given one model.
C.4. Expanded Results
This section presents a broader set of experimental results. We evaluate multiplicity across several datasets using both the BIGSCIENCE TO model fine-tuned with LoRA (see Table 6) and the FLAN-T5 model fine-tuned using the Tfew recipe (see Table 7). Additionally, Figures 7, and 8 visualize the evaluated multiplicity versus the stability measure for Bank, Diabetes, and German Credit datasets respectively. We also report the correlation between the stability measure and various multiplicity evaluation metrics for BIGSCIENCE TO model fine-tuned using Tfew recipe (see Table and FLAN-T5 model fine-tuned using Tfew recipe (see Table 9).
Table 6. Multiplicity Evaluation Metries for Different Datasets and Number of Shots. Evaluated on 40 fine-tuned BIGSCIENCE TO models on LoRA using different random seeds. Multiplicity observed in predictions across different fine-tuned model, even when models exhibit similar accuracy (in this setting d 0.02).
Multiplicity Evaluation Metries (BIGSCIENCE TO)
Dataset
No. Shots
Arbitrariness
Discrepancy
Avg. Pairwise Disagreement
Avg. Pred. Variance
Avg. Pred. Range
Avg. Model Accuracy
64
11%
6%
9%
0.01
0.11
83%
Adult
128
10%
9%
6%
0.01
0.10
84%
512
11%
3%
10%
0.01
0.12
85%
64
19%
10%
6%
0.04
0.40
70%
German
128
17%
11%
6%
0.01
0.16
71%
512
21%
14%
8%
0.03
0.26
72%
64
20%
13%
11%
0.04
0.21
70%
Diabetes
128
16%
14%
11%
0.08
0.14
73%
512
19%
13%
11%
0.04
0.17
76%
64
13%
9%
7%
0.01
0.28
66%
Bank
128
14%
9%
7%
0.03
0.21
73%
$12
14%
8%
7%
0.03
0.22
78%
Table Evaluated Multiplicity for Different Datasets and Number Shots. Evaluated on fine-tuned FLAN-TF models using TYew recipe with different random seeds. Multiplicity observed in predictions across different fine-tuned models, when models exhibit similar accuracy (in this setting 0.02). The accuracy FLAN model on the dataset less than the BIGSCIENCE model observed in Table 1.
Multiplicity Evaluation Metrics (FLAN-TS)
Dataset
Shots
Arbitrariness
Discrepancy
Avg. Pairwise Disagreement
Avg. Pred. Variance
Avg. Pred. Range
Avg. Model Accuracy
64
13.96%
6.93%
5.05%
0.010
0.139
74.25%
Adult
128
8.81%
3.84%
3.39%
0.008
0.091
77.50%
512
12.02%
5.71%
4.49%
0.012
0.123
79.17%
18.50%
11.00%
6.19%
0.015
0.194
64.85%
German
128
30.00%
13.50%
10.47%
0.031
0.287
69.25%
512
35.50%
16.50%
12.88%.
0.041
0.362
69.40%
64
15.58%
7.79%
6.23%
0.016
0.170
68. 68.18%
Diabetes
128
11.69%
5.84%
4.81%
0.012
0.129
59. $9.29%
512
21.43%
9.74%
7.37%
0.022
0.207
69.55%
64
12.86%
7.46%
4.69%
0.003
0.125
66.96%
Bank
128
17.95%
6,90%
6.59%
0,006
0.165
65.94%
512
17.17%
6.61%
6.24%
0.017
0.173
79.40%
18

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
Table This table reports the Spearman correlation between the stability measure, predicted probabilitiie, and the drop-out method with various multiplicity evaluation metrics for different numbers of shots on several datasets (BiGSCIENCE TO fine-tuned using Tfew recipe). most cases, stability measure Sk.e f) shows higher correlation probabilities and drop-out, indicating that the stability measure Sa. (x, f' better informs about the multiplicity than the other measures do. The dropout method performing better than naive predicted probability.
these multiplicity measures compared to predicted
Dataset
Number of Shots
Measure
Arbitrariness Pairwise Disagreement Prediction Variance Prediction Range
Pred. Prob.
0.67
0.66
0,50
0.62
64
Drop-Out
0.83
0.78
0.81
0.87
Stability
0.95
0.90
0.91
0.89
Adult
Pred. Prob.
0.67
0.62
0.30
0.54
128
Drop-Out
0.74
0.83
0.69
0.8]
Stability
0.80
0.96
0.84
0.91
Pred. Prob. 0.70
0.69
0.56
0.72
512
Drop-Out
0.78
0.78
0.88
0.88
Stability
0.90
0.86
0.93
0.92
Pred. Prob.
0.99
0.99
0.80
0.79
64
Drop-Out
0.73
0.71
0.82
0.76
Stability
0.95
0.95
0.98
0.84
German Credit
Pred. Proh.
0.57
0.57
0.86
0.86
128
Drop-Out
0.50
0.56
0.74
0.84
Stability
0.54
0.54
0.87
0.87
Pred. Prob.
0.54
0.56
0.83
0.82
512
Drop-Out
0.69
0.67
0.72
0.65
Stability
0.59
0.60
0.87
0.86
Pred. Prob.
0.03
0.38
0.04
0.08
64
Drop-Out
0.30
0.19
0.54
0.46
Stability
0.45
0.51
0.31
0.23
Diabetes
Pred. Prob.
0.88
0.93
0.93
0.95
128
Drop-Out
0.89
0.92
0.92
0.94
Stability
0.92
0.95
0.93
0.95
Pred. Prob.
0.21
0.23
0.24
0.30
512
Drop-Out
0.74
0.83
0.75
0.74
Stability
0.80
0.89
0.74
0.68
Pred. Prob.
0.70
0.69
0.56
0.74
64
Drop-Out
0.79
0.77
0.77
0.80
Stability
0.83
0.78
0.81
0.80
Bank
Pred. Prob.
0.54
0.57
0.73
0.62
128
Drop-Out
0.62
0.70
0.75
0.51
Stability
0.79
0.84
0.87
0.86
Pred. Proh.
0.71
0.68
0.81
0.76
512
Drop-Out
0.90
0.89
0.87
0.84
Stability
0.91
0.92
0.91
0.87
Pred. Prob.
0.70
0.21
0.30
0.69
64
Drop-Out
0.56
0.48
0.54
0.56
Stability
0.98
0.86
0.98
0.98
Heart
Pred. Prob. 0.61
0.46
0,50
0.26
128
Drop-Out
0.64
0.76
0.74
0.83
Stability
0.89
0.90
0.97
0.87
Pred. Prob. 0.80
0.65
0.48
0.35
512
Drop-Out
0.94
0.90
0.90
0.94
Stability
0.89
0.95
0.86
0.95
Pred. Prob.
0.83
0.83
0.40
0.83
64
Drop-Out
0.85
0.83
0.96
0.97
Stability
0.76
0.69
0.86
0.75
Car
Pred. Prob.
0.56
0.26
0.29
0.01
128
Drop-Out
0.63
0.66
0.57
0.52
Stability
0.97
0.91
0.93
0.94
Pred. Prob.
0.91
0.94
0.72
0.86
512
Drop-Out
0.98
0.96
0.95
0.93
Stability
0.68
0.59
0.56
0.67
19

---

Quantifying Prediction Consistency Under Fine-Tuning Multiplicity in Tabular LLMs
Table 9. This table reports the Spearman correlation between the predicted probabilities, drop-out method, and the stability measure with various multiplicity evaluation metries for different numbers shots several datascts (Flan T5 model fine-tuned using Tfew recipe). In most cases. the stability measure shows higher correlation with these multiplicity measures compared to predicted probabilities and drop-out, indicating that the stability measure better informs about the multiplicity than the other measures do. dropout method performs competitively in some cases.
Dataset
Number Shots Measure
Arbitrariness Pairwise Disagreement Prediction Variance Prediction Range
Pred. Prob. 0.62
0.67
0.72
0.56
64
Drop-Out 0.60
0.65
0.67
0.57
Stability
0.63
0.72
0.72
0.60
Adult
Pred. Proh.
0.75
0.74
0.65
0.75
128
Drop-Out 0.85
0.78
0.75
Stability
0.88
0.90
0.84
0.79
Pred. Prob. 0.78
0.68
0.42
0.45
512
Drop-Out
0.78
0.78
0.42
0.45
Stability
0.79
0.71
0.78
0.68
Pred. Prob.
0.27
0.04
0.27
0.17
64
Drop-Out
0.73
0.45
0.60
0.17
Stability
0.77
0.67
0.78
0.76
German Creait
Pred. Prob.
0.85
0.76
0.85
0.91
128
Drop-Out
0.86
0.91
0.85
0.91
Stability
0.89
0.91
0.89
0.92
Pred. Prob.
0.42
0.29
0.27
0.19
512
Drop-Out Out
0.43
0.36
0.28
0.33
Stability
0.61
0.60
0.67
0.69
Pred. Prob.
0.09
0.04
0.27
0.23
64
Drop-Out
0.24
0.41
0.54
0.50
Stability
0.27
0.55
0.31
0.25
Diabetes
Pred. Proh.
0.16
0.06
0.17
0.16
128
Drop-Out
0.46
0.55
0.54
0.63
Stability
0.52
0.57
0.44
0.52
Pred. Prob.
0.61
0.35
0.12
0.19
512
Drop-Out
0.71
0.42
0.42
0.51
Stability
0.79
0.40
0.39
0.40
Pred. Proh.
0.26
0.04
0.27
0.17
64
Drop-Out
0.24
0.60
0.60
0.60
Stability
0.77
0.67
0.78
0.76
Bank
Pred. Prob.
0.45
0.54
0.73
0.62
128
Drop-Out
0.62
0.70
0.75
0.82
Stability
0.89
0.71
0.78
0.84
Pred. Prob.
0.42
0.29
0.27
0.11
512
Drop-Out
0.44
0.29
0.37
0.43
Stability
0.61
0.60
0.30
0.380
20

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
Prediction Range
0.8
Preciction Variance
Pairwise Disagreemen ennt
0.4
1.00
0.6
Arbitrariness
0.4-
0.050
0.2
0.50
0.025
0.25
0.0
0.00
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
0.0
0.5
1.0
Stability
S(x.
Prediction Probabilities fix)
Drop Pred
Figure Evaluated multiplicity (assessed on 40 retrained models) versus our stability measure (evaluated on one model) for the 512-shot setting the Bank dataset. The plots demonstrate that high stability values correspond low multiplicity across various multiplicity evaluation metrics. Predictive probabilities and Drop- Out Out not providing any providing any useful insight into multiplicity.
0.100
reriise Disagreement
1.00
Prediction Range
Prediction Variance
0.4
Arbiraainess
0.75
0.6
0.4
0.050
0.2
0.50
0.2
0.025
0.25
0.0
0.25
0.50
0.75
0.25
0.50
0.75
0.25 0.50
0.75
0.25 0.50 0.75
Stability
Six, f)
Prediction Probabilities
Pred
Figure 7. Evaluated multiplicity (assessed on 40 retrained models) versus stability measure (evaluated on model) for the 512-shot setting on the Diabetes dataset. The plots demonstrate high stability values correspond to multiplicity across various multiplicity evaluation metrics. Predictive probabilities not providing any providing any useful insight about multiplicity. The drop-out out method performs better than predictive probabilities still worse than stability.
0.06
Painwise Disagreement
1.00
Prediction Range
0.4
M
0.6
Arbitrariness
0.75
0.04
0.3
0.4
Prediction
0.2
0.50
0.02
0.1
0.25
0.00
0.0
0.0
0.5
0.0
0.5
0.0
0.5
0.0
0.5
Stability
Probabilities
Drop Pred
Figure 8. Evaluated multiplicity (assessed on 40 retrained models) versus our stability measure (evaluated on one model) for the 512-shot setting on the German Credit dataset. The plots demonstrate that high stability values correspond to low multiplicity across various multiplicity evaluation metries. In this setting Prediction probability is performing competitively. But generally stability measure provides better insight into the multiplicity of predictions compared to predicted probabilities. drop-out method performing significantly worse than the other two measures.
21

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
Stability Six,
0.06
Pairwise Disagreement
1.00
Prediction Range
Prediction Varaance
0.4
0.04
0.4
0.50
0.2
0.02
0.2
0.25
0.00
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
0.25 0.50 0.75
10"3
10
10
Figure 9. Ablation study on different a values: The chosen value of a = 0.01 yields the best performance across all evaluation metrics. Smaller values of (e.g.. 10-4) result perturbations that close the original data points, leading similar prediction probability alone, as the sampled points are nearly identical. On the other hand, larger values (c.g., 10-3) produce overly noisy perturbations, rendering the results uninformative.
C.5. Expanded Ablations
We include the complete ablation results referenced in the main paper. Table 11 presents the results of varying the sample size Table and Figures examines the effect of different values on the stability measure. Additionally, Table 13 compares the stability measure with the drop-out method with different drop-out rates p. Finally, Table 10 compares the stability measure with variability-basee alternatives. Table 14 compares performance under LoRA, Prompt Tuning, and Prefix Tuning the Bank dataset.
Table 10. Correlation between the proposed consistency measures and various multiplicity evaluation metrics. New additional measures:
Si (x) = (xx) f(x)] (absolute variability). S2(x) = %. (((x)) - fx)) (squared variability). proposed Stability
measure oulperforms both dropout-based method and purely variability-based alternatives
Dataset
Number of Shots
Measure
Arbitrariness
Pairwise Disagreement
Prediction Variance
Prediction Range
Pred. Prob.
0.67
0.62
0.30
0.54
Drop-Out
0.74
0.83
0.69
0.81
(xx
0.70
0.65
0.63
0.72
Adult
128
(x)
0.70
0.64
0.60
0.73
Stability
0.80
0.96
0.84
0.91
Table 11. Ablation study different values: Correlation hetween stability measure (evaluated single model) and various
measures of multiplicity for different sample sizes & on the Di Diabetes dataset (TO model). We observe better performance with increasing suggested by theoretical results. Larger sample size ls values advantageous, they ensure that the guarantees hold with high probability. However, computational cost of model inference (forward pass) increases.
4:
Prediction Range
Prediction Variance
Pairwise Disagreement
Arbitrariness
0.77
0.53
0.52
5
0.82
0.83
0.56
0.55
10
0.87
0.87
0.62
0.61
20
0.89
0.88
0.70
0.79
22

---

Quantifying Prediction Consisteney Under Fine-Tuning Multiplicity in Tabular LLMs
Table Ablation study on different values: Correlation between our stability measure (evaluated on one model) and various evaluation measures for different values of and evaluated multiplicity for Diabstes datasct and 128-shot case (TO model). Best performance observed when 10 10-?. To guide the choice a, one could consider the spread of training data points the embedding space (e.g,, use value equivalent to 10% of the variance of the training data). For all our experiments, which consistently worked well across different datasets and experiments. When a is too small, we basically sample (almost) the same points and stability measure not informative than the prediction probability. When is too large, loses all information about the data point.
used fixed value of 0.01,
Prediction Range
Prediction Variance
Pairwise Disagreement
Arbitrariness
10
0.82
0.83
0.84
0.80
10-3
10
0.91
0.92
0.90
0.86
10-2
10
0.95
0.93
0.92
10
0.10
0.08
0.33
0.23
Table This table reports the correlation between stability measure and various evaluated multiplicity for the 512-shot setting on the Diabetes dataset. The stability measure Sa,o (z, f) shows higher correlation with multiplicity compared to predicted probabilities drop-out and ensemble method, indicating that the stability measure 5 f) better informs multiplicity than the other measures.
Method
Arbitrariness
Pairwise Disagreement
Prediction Variance
Prediction Range
Pred. Prob.
0.21
0.23
0.24
0.30
drop-out p = 0.01
0.23
0.27
0.28
drop-out = 0.1
0.62
0.61
0.59
0.64
drop-out 0.2
0.74
0.36
0.53
0.54
drop-out 0.5
0.16
0.17
0.18
0.16
Stability
0.80
0.89
0.74
0.68
Table 14. We compare the correlation in our default LoRA setting against Prompt Tuning and Prefix Tuning (Bank dataset) to assess the generalizability of our method beyond LoRA. Although the stability measure achieves the highest correlations under LoRA, still provides meaningful signals under Prompt and Prefix tuning.
Dataset
Measure
Arbitrariness
Pairwise Disagreement
Prediction Variance
Prediction Range
Pred. Prob. (LoRA)
0.54
0.57
0.73
0.62
Pred. Prob. (Prompt Tuning)
0.50
0.48
0.61
0.55
Pred. Prob. (Prefix Tuning)
0.52
0.49
0.59
0.51
Drop-Out (LoRA)
0.62
0.70
0.75
0.51
Bank
Drop-Out (Prompt Tuning)
0.48
0.53
0.60
0.49
Drop-Out (Prefix Tuning)
0.55
0.50
0.58
0.46
Stability (LoRA)
0.79
0.84
0.87
0.86
Stability (Prompt Tuning)
0.63
0.60
0.58
0.61
Stability (Prefix Tuning)
0.59
0.62
0.60
0.57
23


Reviews:
[{"review_id": "FqYMwfKlZK", "reviewer": "Reviewer_G1UA", "summary": "The work studies the problem of model multiplicity in LLM classification for tabular data. Model multiplicity refers to the phenomenon that multiple models of similar accuracy assign confliciting predictions to individual instances. The authors propose a measure of model multiplicity (that does not require retraining), provide some theoretical observations about it (Section 3.2), and evaluate it in six tabular datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review!\n\nLink to **PDF** with new figures and tables: https://drive.google.com/file/d/1zMdT0zdMrIPO9eUCHGYZ-WUlYsgbyDYu/view\n\n---\n**Regarding Mean and Variability Term:** Our stability measure relies upon both local variability and mean confidence because they capture synergistic aspects of prediction robustness in classification (see motivational Fig. 8 in PDF). Model prediction confidence has long been used as a measure of reliance (roughly the distance from the decision boundary), e.g., in multi-class classification, if the output logits are [0.4,0.3,0.3] vs [0.6, 0.2, 0.2], the latter may be more robust. If the average confidence in a neighborhood is higher, it is an even stronger indicator of robustness (e.g., a prediction differing significantly from all its neighbors may be unreliable). Points near decision boundaries might exhibit higher arbitrariness and pairwise disagreement than those in high-confidence neighborhoods (for classification). However, the mean confidence alone is not sufficient if the point lies in a highly fluctuating region, e.g., two neighborhoods with identical variability ≈$0.05$ but markedly different mean confidences $0.8$ vs $0.33$: the former is still more likely to be in the same class. While just the variability term quantifies local fluctuations, the mean confidence provides crucial information about base neighborhood prediction confidence. We now include an additional experiment, with the variability term $M_1(x)$ and $M_2(x)$, showing a weaker correlation than our measure (Table. 10 in PDF). \n\n---\n**Choice of Sigma:** While our ablation studies showed that $\\sigma=10^{-2}$ achieved optimal results, we found that nearby values $10^{-3}$ to $10^{-1}$ still maintained strong correlations with multiplicity metrics (Fig. 7), demonstrating tolerance to the exact choice of $\\sigma$. Drawing inspiration from related domains such as kernel methods [3,4] or certified robustness [1,2] which involve a similar neighborhood hyperparameter, we suggest the following **practical, data-driven method to set $\\sigma$ without retraining**:\n\n* Compute Pairwise Distances: For all training samples, calculate the median distance $d_{med}$ between each point and its k-nearest neighbors ($k=5$) in the embedding space.  \n\n* Set $\\sigma$ as a Fraction of $d_{med}$: Choose $\\sigma = 0.1 d_{med}$. This captures the natural scale of the data while ensuring perturbations stay within the local neighborhood.  \n\nWe tested this on Diabetes and Adult datasets. The computed $d_{med}$ was ~0.1, leading to $ \\sigma = 0.01$, which matched the optimal ablation value (Fig. 7). This method generalizes across datasets without retraining.  $ d_{med} $ reflects the inherent data density—perturbations smaller than this preserve local structure, while larger values risk overshooting. Several ML methods use similar dataset statistics e.g., median(pairwise distances) approaches to set parameters (e.g., DBSCAN [4], Kernel Methods / RBF Kernels [3], etc). Importantly, this suggested method for choosing $\\sigma$—based on distance to nearest neighbors in the training set—requires no model retraining and hence scales seamlessly to larger datasets.  We will include this procedure in the revised manuscript.  \n\nRelated fields like certified robustness rely on similar choices of hyperparameter $\\sigma$ between $10^{-2}$ to $10^{-1}$, often guided by dataset scale or empirical tuning [1,2].  We note that our baseline methods such as Drop-out [Hsu et al 2024] also rely on selection of hyperparameters such as the drop-out rate (heuristic or search-based).\n\n--- \n**Larger Dataset:**  We have now included runtime experiments on the Adult dataset (see Fig. 9 and Table 11 in PDF), a dataset which contains over 40k samples and represents a setting where retraining multiple models is computationally expensive. Specifically, evaluating multiplicity by retraining 100 fine-tuned models on this dataset took over 8 hours, and another baseline, AWP, required 16 hours. In contrast, our method (stability) just took 26 mins, since it does not require retraining. Despite a drastic reduction in runtime, our method achieved higher correlation with multiplicity metrics compared to baselines methods. See bar chart in Fig. 9 and detailed training and evaluation runtimes in Table 11 in PDF. Our core strength lies in the fact that our measure, computed using just a single model, demonstrates high correlation with actual multiplicity which is computationally expensive as it requires fine-tuning several models. \n\n---\n**On Figure 3:** We kindly refer to our response to Reviewer dw7t under **\"Clarity on Figure 3\"**.\n\n---\n1 Cohen et al, Certified Adversarial Robustness via Randomized Smoothing\n\n2 Salman et al, Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers\n\n3 Cortes et al, Support-vector networks\n\n4 Ester et al, A density-based algorithm for discovering clusters in large spatial databases with noise", "claims_and_evidence": "The proposed metric (called Stability) implicitly assumes that local neighborhoods in embedding space reflect nearby models in function space.\nThis is the key insight that allows the authors to estimate model multiplicity without retraining. \n\nWhile I see no a priori reason why this should be true or false, the high Spearman correlation between Stability (which does not require retraining) and metrics such as Arbitrariness and Prediction Variance (which do require expensive retraining) is evidence that this assumption is true. That Stability has a higher Spearman correlation than other metrics that do not require retraining (e.g. Drop-Out) is a major strength and the main contribution of this work.", "methods_and_evaluation_criteria": "The exact choice for the metric (Stability) confuses me in a few ways. It is worth explaining how this metric is computed, as in my opinion it is not intuitive. For a given label $x$, we assume that $f(x) = [f_1(x), \\ldots, f_C(x)]$ outputs the softmax over classes, and that we predict that label $x$ belongs to class $c = \\text{argmax}_{i \\in [C]} f_i(x)$. \n\nThe Stability metric only looks at the predicted probability $f_{c}(x)$, and ignores all the other probabilities in the softmax. The metric is \n\n$\\frac{1}{k} \\sum_{x_i \\in N_{x,k}} f_c(x_i) - \\frac{1}{k} \\sum_{x_i \\in N_{x,k}} |f_c(x_i) - f_c(x)|$. We see that the Stability metric has two contributions: a Local Averaging term and a Variability Penalization term. I do not understand why the Local Averaging term is needed. We already know that $f_c(x)$ is the largest predicted probability out of all the classes, so why do we care about its magnitude in a neighborhood around $x$? I would find just the Variability Penalization term to be a more intuitive metric, since my main concern is how much the value of $f_c(x)$ changes within a neighborhood of $x$. The magnitude does not matter because we already predicted class $c$. \n\nMy main question ( which I will repeat in the Questions Section) to the authors is why they did not consider a metric such as: \n\n$$M_1(x) =  \\frac{1}{k} \\sum_{x_i \\in N_{x,k}} |f_c(x_i) - f_c(x)|$$\n\nor its square, so that this has the interpretation of a variance: \n\n$$M_2(x) =  \\frac{1}{k} \\sum_{x_i \\in N_{x,k}} (f_c(x_i) - f_c(x))^2.$$\n\nWhat exactly does the metric $S$ capture that $M_1$ or $M_2$ miss? I also find the metric $M_2$ to be particularly pleasing since it looks very similar to the Predictive Variance metric used in the paper. Moreover, assuming the experiments do not take too long, I am curious if metrics such as $M_1$ and $M_2$ also exhibit correlation with the more expensive metrics that require retraining.", "theoretical_claims": "I checked the proof of Theorem 3.3 and verified its correctness.", "experimental_designs_or_analyses": "I think authors need to explain more clearly what exactly Figure 3 is plotting, namely how exactly the uncertainties in Figure 3 are computed. I may be mistaken, but from my understanding, the Stability metric is evaluated across the entire test dataset for a fixed model $f$, i.e.  $[S(x_1, f), S(x_2, f), \\ldots S(x_N, f)]$. Then (using Fig 3a as an example), the predictive range is evaluated across the entire test dataset for the 40 trained models $[\\text{PR}\\_{\\delta}(x_1), \\ldots, \\text{PR}_{\\delta}(x_N)]$. What we see in Figure 3a is these two arrays plotted together. If my understanding is correct,where does the uncertainty come from? Is $S(x, f)$ evaluated for many different functions $f$? \n\nI also had a question (which I repeat in the Questions section) about Table 1, namely what exactly \"Avg. Prediction Range\" and \"Avg. Prediction Variance\" means. The prediction range and prediction variance are a function of the embedding $x$ and the class prediction $c$, and it is unclear how authors averaged over these two quantities. I assume authors averaged over the dataset, using $c = \\text{argmax}_{i \\in [C]} f_i(x)$ for each embedding $x$, however this is not explicitly stated.", "relation_to_broader_scientific_literature": "See my answer in \"Claims And Evidence\".", "essential_references_not_discussed": "I do not see any missing related works in this paper.", "other_strengths_and_weaknesses": "One of the most intriguing plots in this paper is Figure 7 in the Appendix. This figure clearly shows that the parameter $\\sigma$ (which controls how large of a neighborhood around $x$ you perturb) is the single most important parameter to tune to get the strong correlations observed in this work. Indeed, choosing $\\sigma = 10^{-2}$ clearly makes the Stability metric correlated with all the other metrics that require retraining. At the same time, this correlation disappears if sigma is either too big or too small. In practice, however, one does not have the luxury of comparing against metrics that require retraining to tune $\\sigma$. How do authors recommend $\\sigma$ be chosen in practice? There is one offhand remark that \"To guide the choice of $\\sigma$, one could consider the spread of training samples\", however this is not explored further. I think this is one of the most important problems that authors should address, since if $\\sigma$ can be easily estimated from the training data, then this approach can and should be used in practice by all high stakes LLMs to identify problematic points in the test set. But if $\\sigma$ cannot be estimated without retraining, then this approach is not useful in practice. \n\nAssuming the above is resolved, I think the paper would be much stronger if authors used the Stability metric on a larger dataset where retraining 40 times is computationally infeasible (require days/weeks of fine tuning). Authors could use the Stability metric to identify which points on this larger dataset could be more susceptible to arbitrary decisions. Perhaps these are data-points that belong to a minority group, or are mislabeled?", "other_comments_or_suggestions": "I have no further suggestions. I am leaning towards reject, as I think the paper needs some polishing and more results, particularly on a larger datasets, before it is ready for acceptance at ICML.", "questions_for_authors": "1. (See Methods And Evaluation Criteria for context) Why choose the Stability metric over metrics such as $M_1$ and $M_2$? Do metrics such as $M_1$ and $M_2$ also exhibit the same correlations seen with Stability? \n\n2. (See Experimental Designs Or Analyses for context) In Table 1, what exactly is \"Avg. Prediction Range\" and \"Avg. Prediction Variance\"?\n\n3. (See Other Strengths And Weaknesses for more context) How do authors recommend $\\sigma$ be chosen in practice? Could this be applied to a larger dataset where retraining is infeasible?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ha7T7VEFtE", "reviewer": "Reviewer_ZkiU", "summary": "This paper studies the problem of fine-tuning multiplicity in tabular LLMs, where models trained from the same pre-trained checkpoint under different conditions (e.g. random seeds) make inconsistent predictions on the same inputs. The authors propose a new measure called consistency to estimate the robustness of individual predictions without retraining multiple models. This is done by perturbing the input in the embedding space and computing the average output behavior. The paper provides a theoretical guarantee that high consistency implies stability across a broad class of similarly performing fine-tuned models. Empirical results on six tabular datasets show that the proposed score correlates well with several multiplicity metrics. The method is also compared to prediction confidence and a dropout-based baseline, and performs better in most cases.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their positive review!\n\nLink to PDF with new figures and tables: https://drive.google.com/file/d/1zMdT0zdMrIPO9eUCHGYZ-WUlYsgbyDYu/view\n\n---\n**Regarding downstream use**: While our work focuses on quantifying the stability of predictions—to our knowledge, the first to do so for LLMs in the context of fine-tuning multiplicity—this is a necessary and foundational step for enabling trust in downstream applications. We agree that practical uses are important, and our measure would support actionable decisions: practitioners can filter out or exercise caution for the low-stability predictions, e.g., a candidate getting different decisions from similar models in a loan decision will cause reputational risk or even fairness concerns. However, they can trust predictions on data-points with high stability, which are provably robust across fine-tuned variants without actually retraining an ensemble of models. This aligns with deployment needs in high-stakes domains such as healthcare, finance, hiring, and education, where reliability often matters more than full coverage. Using stability scores to actively mitigate fine-tuning instability is an exciting direction for future work. However, our current goal is to rigorously define and validate a stability measure, both theoretically and empirically. These results lay the necessary groundwork for future methods that could leverage stability scores to guide or regularize training using more stable data points. We will add a brief discussion on these directions in the paper.\n\n---\n**Scope of model class (W2):** Our focus on seed-induced variations aligns with prior work on predictive multiplicity [1,2,3,4], where controlled stochasticity (e.g., initialization, data shuffling) is used to isolate the impact of training randomness—a foundational and prevalent source of instability in fine-tuning. While practical deployments may encounter other sources of variation (e.g., hyperparameter changes or distribution shifts), we deliberately focus on this aspect to **highlight** that even small sources of randomness can lead to highly arbitrary predictions, as demonstrated in our experiments. Several related works on multiplicity also adopt this setting [1,2,3,4].\n\n---\n**Motivation for multiplicity measures in Section 2 (Q1):**  We will revise the text to better clarify the role of each measure. Briefly, we include these standard multiplicity metrics—Arbitrariness, Discrepancy, Pairwise Disagreement, Prediction Variance, and Range—as they each capture different facets of prediction inconsistency across fine-tuned models. These measures serve as ground truth evaluations of fine-tuning multiplicity and allow us to benchmark the effectiveness of our proposed stability score in predicting multiplicity without retraining. Arbitrariness and Discrepancy capture label-level disagreement; Prediction Variance and Range capture the spread of softmax outputs; Pairwise Disagreement captures disagreement among all model pairs. Together, they provide a comprehensive picture of multiplicity and motivate our evaluation framework.\n\n---\n**Novelty of the stability measure (Def 3.1):**\nWhile our formulation shares surface-level similarities with local robustness metrics (e.g., certified robustness uses the mean of predictions in a neighborhood), our objective and motivation are fundamentally different. Traditional robustness measures typically try to capture how resistant a model is to worst-case perturbations (e.g., adversarial examples), whereas our **stability measure tries to capture prediction consistency across a class of fine-tuned models**—a fundamentally different notion that we refer to as fine-tuning multiplicity. To the best of our knowledge, no prior work has proposed this exact formulation to quantify robustness to fine-tuning variability in LLMs or tabular settings. Unlike adversarial robustness, we sample random local perturbations in the embedding space to estimate prediction smoothness in the neighborhood, combining local confidence and variability to derive a **probabilistically guaranteed lower bound** (Thm 3.3) on prediction consistency. We will clarify this distinction in the revised draft.\n\n---\n**Theoretical Results:** Due to character limit, we kindly refer to our response to Reviewer dw7t under **\"Theoretical Results\"**.\n\n---\n[1] Gomez, et al. Algorithmic arbitrariness in content moderation\n\n[2] Hsu, H. et al.. Rashomon capacity: A metric for predictive multiplicity in classification\n\n[3] Watson-Daniels, et al. Predictive multiplicity in probabilistic classification\n\n[4] Hsu, H., et al Dropout-based rashomon set exploration for efficient predictive multiplicity estimation.", "claims_and_evidence": "The central claims -- that fine-tuning multiplicity is a real issue in tabular LLMs and that the proposed consistency score captures prediction robustness -- are mostly supported by solid evidence. The authors provide empirical results across multiple datasets, models, and tuning setups, and compare their method to reasonable baselines. The claim that high consistency implies stability is backed by a theoretical guarantee, though it relies on strong assumptions that are not verified or estimated in practice. The paper also shows strong correlations between the consistency measure and multiplicity metrics, but stops short of demonstrating how this score can be used in practice (e.g. for model selection or filtering). As a result, the measure remains primarily diagnostic, and the practical utility of the score is not fully established.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The main theoretical result provides a lower bound on the prediction of a fine-tuned model in terms of the proposed consistency score. The proof appears correct and is clearly presented. \n\nHowever, the bound relies on constants like alpha, t, and L that are not estimated or discussed empirically, which makes it difficult to apply in practice. The key assumption about the stochasticity of the model class is also strong and not verifiable. Plus, the guarantee is one-sided as it only certifies that predictions won't fall too far below the estimated consistency score, but provides no insight for cases where consistency is low and predictions may still be reliable. These choices are acknowledged by the authors, but they constrain how actionable the theoretical results are in practice.", "experimental_designs_or_analyses": "The experimental setup is sound and addresses the core claims of the paper. The authors evaluate their method across multiple datasets, models (T0 and FLAN-T5), and tuning strategies (T-Few and LoRA), and use standard multiplicity metrics. They also include ablation studies for key hyperparameters. However, the experiments focus only on correlation with multiplicity and do not explore downstream uses of the consistency score, such as improving model reliability or filtering unstable predictions. This limits the practical insight gained from the analysis.", "relation_to_broader_scientific_literature": "The paper connects well with existing work on predictive multiplicity, particularly the Rashomon effect and related metrics like arbitrariness and disagreement. It builds on recent methods for measuring multiplicity in neural networks and extends them to tabular LLMs, which are less explored. It also draws from prior work on robustness via perturbations and embedding space sampling.", "essential_references_not_discussed": "No major omissions stood out", "other_strengths_and_weaknesses": "Strengths\n1. The paper introduces a simple and computationally efficient method to estimate prediction stability without retraining, which is well-motivated for tabular LLMs.\n2. Theoretical analysis (Theorem 3.3) is clear and adds value, even if assumptions are satiesfied.\n3. Evaluation is thorough, which covers multiple datasets, models, and fine-tuning methods.\n\nWeaknesses:\n1. The consistency measure is only evaluated in terms of correlation with multiplicity metrics -- its downstream usefulness remains unclear.\n2. The model class considered is limited to variations from random seeds and does not address other practical sources of instability.\n3. No methods are proposed to reduce or mitigate multiplicity; the work is diagnostic only.\n4. Synthetic experiments (Figure 2) are illustrative but not well-analyzed or connected to real-world behavior.\n5. Assumptions behind the theoretical guarantee are strong and cannot be verified empirically; constants in the bound are not grounded.", "other_comments_or_suggestions": "1. Figure 2 does not make sense in the context of the paper and should be motivated or better connected to the rest of the paper.\n2. The authors should consider moving “Candidate Measure: Prediction confidence” from Section 3 to another subsection of Section 3.\n3. The paper would benefit from a brief discussion of how the proposed measure could be used in downstream tasks", "questions_for_authors": "1. Can authors provide more context about how/why each of the measures mentioned in Section 2 is useful, and why they are used in the context of this paper?\n2. How is stability (Definition 3.1) different from other similar measures? I can think of some sort of adversarial robustness metric in the embedding space off the top of my head. Are there any prior works that have considered this measure? It could be in the context of other applications.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Pgn2gPt0Ic", "reviewer": "Reviewer_dw7t", "summary": "This paper addresses the challenge of ​fine-tuning multiplicity in tabular LLMs, where minor variations in training (e.g., seeds, hyperparameters) lead to conflicting predictions across equally performant models. The authors propose a ​stability measure, which quantifies prediction robustness by analyzing the local behavior of a single model around an input in the embedding space.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review!\n\nLink to **PDF** with new figures and tables: https://drive.google.com/file/d/1zMdT0zdMrIPO9eUCHGYZ-WUlYsgbyDYu/view\n\n---\n\n**Regarding strong assumptions:** Our theoretical analysis draws inspiration from standard assumptions in optimization and statistical learning (e.g., bounded gradients, unbiased estimators) to derive tractable mathematical guarantees. While these assumptions may not hold exactly in practice, they are common in theoretical ML literature and serve to formalize the intuition and justification behind our stability measure. Crucially, our experiments demonstrate that the measure remains effective even when these assumptions are relaxed (e.g., under non-LoRA fine-tuning like T-Few), suggesting robustness to mild violations. The bounded gradient assumption aligns with parameter-efficient methods like LoRA, where low-rank updates naturally constrain function space deviations [1]. For large distribution shift, our guarantees would not directly apply, which we acknowledge as a limitation. However, our focus is on fine-tuning multiplicity (minor model variations) rather than large distributional changes which makes these assumptions reasonable in context. \nOur core highlights are: (i) Even for small changes to the hyperparameters, fine-tuned models do exhibit a significant amount of multiplicity; and (ii) Our measure, computed using just a single model, demonstrates ~$90$% correlation with actual multiplicity which is computationally expensive as it requires fine-tuning several models, which is our core strength. Also see (Fig. 9 and Table 11) in PDF link for a time comparison.\n\nFuture work could extend this analysis to broader settings. Our method can be implemented by a practitioner without a detailed understanding of this theory. This flexibility allows our approach to be accessible to practitioners, making it a valuable tool for the community. Providing theoretical insights and motivations should be seen as an additional strength—offering deeper understanding and interpretability—rather than a drawback, and we believe such contributions should be encouraged in research. \n\n---\n\n**Theoretical Results:** Our theoretical analysis leverages standard assumptions in optimization and statistical learning (e.g., bounded gradients, unbiased estimators) to derive tractable mathematical guarantees. While these assumptions may not hold exactly in practice, they are common in theoretical ML literature and serve to formalize the intuition and justification behind our stability measure. Crucially, our **empirical results validate the practical utility of the stability measure** across multiple datasets, model architectures, and fine-tuning recipes, demonstrating strong correlation between stability scores and evaluated multiplicity *without requiring explicit assumption or estimation of theoretical constants*. \n\nThe one-sided guarantee is purposeful because our goal is to **certify when predictions are stable**. Specifically, our theoretical result ensures that if the stability score $S(x,f)$ is high, then with high probability the true prediction $F(x)$ will be at least $S(x, f) - \\epsilon$. **Our proposed measure serves as a useful and informative lower bound of the model predictions $F(x)$ with a certifiably small gap**. That is, the prediction won’t fall below this threshold across well-performing fine-tuned models. This is exactly the kind of assurance we want: if the stability score is high, the prediction is robust. \n\n---\n\n**Clarity on Figure 3:** The goal of Figure 3 is to visually show that high stability corresponds to lower multiplicity (aligning with the intuition in Thm. 3.3). We group test datapoints by their stability scores (taking a small “window” of stability values at a time along x-axis). Then, we plot the mean evaluated multiplicity (e.g., prediction range) of those datapoints, along with the standard deviation across them. Sliding this window across the entire test set shows the relationship between stability and multiplicity. This is visually neater than a scatter plot. As for Avg. Prediction Range and Avg. Prediction Variance in Table 1, they are indeed the averages over all datapoints in the test set, e.g., $\\frac{1}{N} \\sum_{i=1}^N  PR(x_i)$. We will clarify these details in the final version. \n\n---\n[1]  Zeng, Y. et.al.. The expressive power of low-rank adaptation.", "claims_and_evidence": "Theoretical guarantees are provided to link high stability scores to robustness across a broad class of fine-tuned models.", "methods_and_evaluation_criteria": "Extensive experiments on real-world datasets (e.g., Diabetes, German Credit) demonstrate that the proposed measure outperforms baselines like prediction confidence and dropout-based methods in capturing fine-tuning multiplicity.", "theoretical_claims": "The probabilistic guarantee (Theorem 3.3) under LoRA-based fine-tuning assumptions provides a principled foundation for the stability measure. The connection between local embedding behavior and model robustness is well-motivated and theoretically justified.", "experimental_designs_or_analyses": "Experiments span multiple datasets (tabular and synthetic), model architectures, and fine-tuning methods. The correlation analysis with multiplicity metrics (e.g., Prediction Variance) convincingly demonstrates the measure’s practical utility.", "relation_to_broader_scientific_literature": "NA", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Weakness:\n\nThe guarantees rely on strong assumptions (e.g., unbiased estimators, bounded gradients). Real-world fine-tuning may violate these, especially with non-LoRA methods or large distribution shifts. The paper does not empirically validate these assumptions or discuss their practical relevance.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wzQ9pTePap", "reviewer": "Reviewer_3rKa", "summary": "The paper proposes a proxy measure of multiplicity defined as the difference in average prediction of the model on a hypersphere of radius $\\sigma$ around the point $x$ and the mean absolute difference between predictions on the hypersphere and the point $x$. The paper shows that this measure strongly correlates with multiple other measures of multiplicity over the randomness of LoRA finetuning, but without the need to retrain models.\n\n## Update after rebuttal\n\nThe rebuttal has largely addressed the concerns of my initial review, thus I have increased my score. I still would not recommend naming the concept as either \"stability\" or \"consistency\", as the latter has also been used in the literature on predictive multiplicity to mean something else. Consider something technically descriptive along the lines of \"Confidence-Variability Discrepancy\" or \"Local Confidence Consistency\", which does not imply immediate interpretation as prediction stability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their review!\n\nLink to **PDF** with new Figures and Tables: https://drive.google.com/file/d/1zMdT0zdMrIPO9eUCHGYZ-WUlYsgbyDYu/view\n\n---\n\n**Additional Re-trainings:** We have included an additional experiment with 100 retrainings (see Table 11 in PDF) and still see a high correlation with our Stability measure. Our core strength lies in the fact that the Stability measure, computed using just a single model, demonstrates ~90% correlation with actual multiplicity which is computationally expensive as it requires fine-tuning several models. We would like to respectfully point out that the related work [5] cited by the reviewer trains thousands of models just for logistic regression. They only re-train 50 models for neural networks on CIFAR-10 citing computational constraints. In our case, this challenge is amplified as we work with LLMs with millions of parameters. Not only is training expensive, but the cost of inference for evaluation also scales with $O(N×m)$, where $N$ is the size of the test set and $m$ is the number of retrained models, making experiments with thousands of models especially demanding in our setting. We will definitely cite this paper and include it in our discussion. \n\n---\n\n**Choice of Sigma:** Due to character limits, we kindly refer to our response to Reviewer G1UA under **\"Choice of Sigma\"**.\n\n---\n\n**Regarding Applicability Outside LoRA Method:** We focus on LoRA since it is currently the most widely adopted parameter-efficient fine-tuning method for LLMs. However, we have now added an ablation using Prompt Tuning and Prefix Tuning (see Table 12 in PDF). While our stability measure continues to correlate with multiplicity under these settings, the correlations are weaker—likely due to the limitations of these tuning methods, which are known to be less effective than LoRA in few-shot scenarios. We will include an ablation with other methods (e.g. full fine-tuning) in the revised version to study the limits of our method.\n\n---\n\n**Adversarial weight perturbation comparison:**  We will definitely cite and elaborate the discussion on AWP in the seminal paper “Rashomon Capacity.” We have *included a new experiment, implementing our adaptation for AWP for LLMs* (as another baseline) since that paper is not tailored for LLMs. While AWP is computationally cheaper than retraining for small models [6], we found it prohibitively expensive for LLMs due to the following reason: Each AWP gradient optimization step requires full forward passes on the test set to make sure the model is in the Rashomon set, contributing to high inference costs alongside high gradient-computation costs. Our adapted AWP implementation for LLMs (16 Hrs) (see Table 11) actually took longer than training multiple models (8 Hrs) while still demonstrating weaker or similar correlation to actual multiplicity, making our stability measure (26 mins) more practical for large-scale deployment. See Fig. 9 & Table 11 in PDF link for results including a time comparison that we will also include in our final version.\n\n---\n\n**Lack of operational meaning in terms of multiplicity:** Due to character limit, we kindly refer to our response to Reviewer G1UA under **Regarding Mean and Variability Term**. \n\n---\n\n**Reviewers Suggestions:** Thank you for the thoughtful suggestions. We will incorporate these into the revised version of the paper. We also appreciate the naming feedback and are actively considering renaming \"*stability*\" to \"*consistency*\" to better reflect the interpretation provided in Theorem 3.3.\n\n---\n\n**Thanks for your questions!**\n\n* Missing Bold Highlights in Table 5: We'll correct this in the revised version.\n\n* X-axis in Figure 3: The x-axis corresponds to the respective metric being plotted—Stability (blue), Prediction Probabilities (green), and Drop Pred (orange). Each curve shows how evaluated multiplicity changes as the value of that stability metric changes, with shaded regions denoting variance. To get the plot, we first group test datapoints by their stability scores (e.g., taking a small “window” of stability values at a time), then we plot the mean evaluated multiplicity (e.g., prediction range) of those datapoints, along with the standard deviation across them.\n\n* Correlation Results Beyond LoRA: Please refer to our response under **Regarding Applicability Outside LoRA Method**.\n\n---\n\n[1] Cohen et al., 2019  Certified Adversarial Robustness via Randomized Smoothing\n\n[2] Salman et al., 2019 Provably Robust Deep Learning via Adversarially Trained Smoothed Classifiers\n\n[3] Cortes, Corinna, and Vladimir Vapnik. \"Support-vector networks.\n\n[4] Ester, Martin, et al. \"A density-based algorithm for discovering clusters in large spatial databases with noise.\n\n[5] Bogdan Kulynych, et.al., Arbitrary Decisions are a Hidden Cost of Differentially Private Training\n\n[6] H Hsu, et al, Rashomon capacity: A metric for predictive multiplicity in classification", "claims_and_evidence": "The paper has two main claims. First, that the proposed measure provides a high-probability lower bound on the confidence of the prediction  of a finetuned model. This claim comes with a proof and depends on a set of restrictive assumptions, but is not directly evaluated. It would be helpful to evaluate the claim on a toy dataset, e.g., in its in-expectation version which would be easier to estimate in practice.\n\nThe second core claim is that the proposed stability measure highly correlates with most other multiplicity measures without requiring to retrain models. This is supported by an empirical comparison to a set of existing metrics, when using 40 re-trainings. However, the paper lacks a comparison to adversarial weight perturbation (see essential references for more details) which, although is more expensive than inference, is still significantly cheaper than retraining.", "methods_and_evaluation_criteria": "The method and experimental settings are reasonable and appropriate. However, as the proposed measure does not have a clear operational meaning (see other strengths and weaknesses for details), and seems only useful insofar it correlates with other operational measures, an important question that is not answered by the paper is whether the metric is applicable beyond LoRA. Thus, it would be helpful to see the whether the metric is useful outside of LoRA, e.g., full finetuning or other adapter methods.", "theoretical_claims": "I have not carefully checked the proof of the formal statement.", "experimental_designs_or_analyses": "40 re-trainings seems quite low for multiplicity measures based on retraining, so it is likely that the numbers have very high variance. It would be helpful to have more re-trainings at least for some settings. [Recent work](https://arxiv.org/abs/2302.14517) suggests in the order of thousands re-trainings is needed to estimate, e.g., prediction variance and disagreement accurately.", "relation_to_broader_scientific_literature": "Measuring multiplicity is an interesting problem, but it can be computationally challenging for realistic models. The paper proposes a method based on sampling a series of predictions in a neighbourhood around a point as a proxy measure that could flag high multiplicity in a way that is computationally inexpensive. However, the proposed method seems to be quite limited in scope, operational interpretation, and have limited applicability in practice due to the dependence on the arbitrary parameter $\\sigma$.", "essential_references_not_discussed": "- The discussion of the [Rashomon capacity](https://arxiv.org/abs/2206.01295) is incomplete. L418 seems to suggest that the methods proposed therein rely on retraining, but they rely on adversarial weight perturbation, which is significantly computationally cheaper than retraining. The paper lacks comparison to an AWP based evaluation of either prediction range or capacity metric.\n- Prediction variance is not a new metric as L179 seems to suggest. It was previously [studied](https://arxiv.org/abs/2302.14517) and shown to be proportional to pairwise disagreement in the case that the hard class predictions are used.", "other_strengths_and_weaknesses": "The proposed method, although significantly less expensive to compute than retraining-based measures of multiplicity, has significant weaknesses:\n- Arbitrary choice of $\\sigma$. First, it seems that there is no principled way to pick $\\sigma$. The choice will likely depend on the dataset, and assuming that the use case of the measure is not to evaluate other standard metrics, then we cannot use other measures as benchmarks to select $\\sigma$. How to choose $\\sigma$ in practice? E.g., there exist [principled ways to choose the Rashomon set parameter](https://www.researchgate.net/profile/Lucas-Paes-7/publication/373307229_On_the_Inevitability_of_the_Rashomon_Effect/links/671185ff069cb92a811a550e/On-the-Inevitability-of-the-Rashomon-Effect.pdf).\n- Lack of operational meaning in terms of multiplicity. Other than the fact that the proposed measure correlates with other standard measures of multiplicity, it does not have a multiplicity interpretation. The interpretation in Theorem 3.3 relies on strong assumption and it is unclear if $\\epsilon'$ can make it meaningful in practice.\n- Limited scope of applicability. It is unclear whether the method is only useful for LoRA or other models as well.", "other_comments_or_suggestions": "- Table 6 in the appendix has the method rows in a different order than the other tables (stability is last)\n- Stability is used for distinctly different notions in the literature on multiplicity and learning theory. I would suggest picking a different name, especially considering that the intepretation (Theorem 3.3) is not really stability but rather something like a test for high confidence.\n- The focus on Tabular LLM finetuning seems a bit odd. The proposed methods should be applicable to other standard settings such as evaluating multiplicity for question answering (think MedMCQA) after LoRA finetuning.\n- It would be helpful to expand to break down the \"assumption\" into an actual list of assumptions, e.g., unbiasedness, bounded Hessian, etc, with a clear justification for each.\n- Watson-Daniels is not cited for prediction range in L140.", "questions_for_authors": "- Why multiple rows (e.g., for the Car dataset) in Table 5 lack the bold highlight?\n- What is the x axis in Figure 3 and other similar figures in the appendix?\n- Do the correlation results hold outside of LoRA finetuning? E.g., other adapters or full finetuning.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "There is no need for a full ethics review, but I would just like to point out that the Diabetes dataset has been withdrawn, and is not recommended for non-diabetes research."}}]

Author rebuttal:

Meta-review: The paper addresses the timely problem of LLM multiplicity for tabular data. All reviewers were ultimately content with the paper and the rebuttal responses provided by the authors. including the soundness of the proposed approach and strength of the experimental results.  There are several additional suggestions that would be good to incorporate in the final version.
Decision: Accept (poster)