Adapting Language Models to Produce Good Class Probabilities for Classification Tasks
Authors: Lautaro Estienne
Matias Vera
Elizabeth Fons
Elena Kochkina
Pablo Piantanida
Luciana Ferrer
Venue: Accepted by TMLR ()

Abstract: Large generative language models (GLM) provide a versatile tool for solving a wide variety of natural processing tasks. GLM responses, though, are provided in the form of text, without an indication of the model's confidence in the answer. This limits the usability of these models on high-risk applications where decisions made based on an incorrect answer can have severe consequences. In this work, we focus on the problem of generating class posterior distributions for text classification tasks like sentiment, news category and intent classification. These posteriors can be used for decision making and as interpretable scores for the user. We show that the naive approach for computing posteriors based on the token posteriors produced by the GLM results in extremely poor posteriors. We then explore different adaptation approaches for improving the quality of posteriors, focusing on low resource scenarios where a small amount of data is available for adaptation. We show that parameter-efficient supervised fine-tuning (SFT), while providing large gains in terms of decision quality, produces suboptimal posteriors due to overfitting. To address this problem, we propose an approach that combines SFT and post-hoc calibration (PHC) using a three-stage training strategy, improving the quality of both posteriors and categorical decisions.

Published in Transactions on Machine Learning Research (01/2026)
Adapting Language Models to Produce Good Class Probabilities for Classification Tasks
Lautaro Estienne
lestienne@fi.c ubo.ar
School of Engineering, UBA, Argentina ICC, CONICET-Uniee
de Buenos Aires, Argentina
LISN, CNRS, Université Paris-Suclay, France International Laboratory on Learning Systems, Canada
Matías Vera
muero uba.ar
School of Engineering, UBA, Argentina CSC, CONICET, Aryentina
Elizabeth Fons
Secabeth.fons@ijpmorry
J.P. Morgan AI Research, UK.
Elena Kochkina
dlena.kochkia @oaecbiina@jmmmmggnnccom
J.P. Morgan Al Research, UK.
Pablo Piantanida
pablo. Sabbooppenttnnidaeeerra.
International Laboratory on Learning Systems, Canada Mila Quebec Al institute, Canada
CNRS Université Paris-Saclay, France
Luciana Ferrer
(ferrer &dc. uba.ar
ICC, CONICET- Universidad Buenos Aires, Angentina
Reviewed on OpenReview: https: openreview. net/ forum? id= WnnII669GR
Abstract
Large generative language models (GLM) provide a versatile tool for solving wide variety of natural processing tasks. GLM responses, though, are provided in the form of text, without an indication the model's confidence the answer. This limits the usability of these models on high-risk applications where decisions made basod on an incorrect answer have severe consequences. In this work, we focus the problem of generating class posterior distributions for text classification tasks like sentiment, news category and intent classification. These posteriors can be used for decision making and as interpretable scores for the user. We show that the naive approach for computing posteriors based on the token posteriors produced by the GLM results in extremely poor posteriors. We then explore different adaptation approaches for improving the quality of posteriors, focusing on low resource scenarios where small amount of data is available for adaptation. We show that parameter-efficient supervised fine-tuning (SFT), while providing large gains in terms of decision quality, produces suboptimal posteriors due to overfitting. To address this problem, propose approach that combines SFT and post-hoc calibration (PHC) using three- stage training strategy, improving the quality of both posteriors and categorical decisions.
1
Introduction
Modern neural language models provide an effective, adaptable and scalable tool for solving many natural language processing tasks. In particular, large generative language models (GLMss like LLaMA 3 (Grattafiori

---

Published in Transactions on Machine Learning Research (01/2026)
Qwen 2.5 (Yang et al. 2024 Phi-4 (Abdin el. 2024]. are currently being used for variety of complex natural language understanding tasks, showing outstanding performance on benchmarks related to reading comprehension, summarization, information retrieval, and generative
et al.
2024..
(Narayan
ict al.
2018 Zellers et al. 2019 Khattab et al. 2023 Omar et al. 2023 OpenAI et al.) 2024 Wei et all
2021p. While surprisingly good out-of-the-box performance can often be achieved with these models, they are
generally unable to produce reliable measures of uncertainty for their answers, as shown in large number of works (Lin et a 2024; Cole et al., 2023: Estienne al. 2023 Kuhn et al., 2022; Zhao et al. 2021]. This
limits the usability of these models for high-risk tasks where only very high confidence answers should be
accepted (Zhang et al. 2021 Yoon et al. 2020
In classification tasks, which are the focus of this work, optimal decisions are made based on class posterior probabilities using Bayes decision theory (Hastie et al., 2001 Bishop 2007 Ferrer 2024). For example, for the common case where all errors are assumed to cost the same, Bayes decisions correspond to the class with the highest posterior. The quality of the decisions then directly depends on the quality of the posteriors. Further, the posterior for the selected class is a measure of the confidence with which the system made the decision. This confidence score is valuable information that can be provided to the end user along with the decision. The goal of this work is to develop an approach for adapting GLMs for downstream classification task to produce good-quality posterior probabilities for decision making and for interpretation by the end user. We use the phrase "good-quality posteriors", to refer to posteriors that can be trusted for decision making with Bayes decision theory (see, for example, Winkler & Murphy (1668)) ). A system's ''s posteriors are better quality than another's if they result in better Bayes decisions.
Various approaches have been proposed for adapting GLMs to a specific domain. Standard techniques such adding examples in the prompt to illustrate how the task should be solved or asking for a detailed explanation before producing an output are part of the family of methods called prompt engineering (or prompting, for short) (Liu et al.) 2023). These methods have relatively low computational cost during the development phase because they do not require updating the parameters of the model. On the other hand, during deployment, they have the disadvantage of consuming tokens from the available prompt. buffer and slowing down inference. Furthermore, the performance of the resulting system has been shown to strongly depend the specifics of the prompt, making it somewhat fragile approach (Zhao et al., 2021 Estienne et al., 2023 Another alternative for adapting GLM to downstream task domain is supervised fine-tuning (SFT), which requires annotated data. for the domain of interest. Due to the large size of modern GLMMs, updating all of their parameters is computationally expensive and requires large amounts of adaptation data. To address this issue, a varioty of sarameter-cffiieent finc-tuning (PEFT) methods were proposed in the literature, among which LoRA (Edward J Hu el al., 2022 is currently the de-facto slandard.
Although SFT methods are very effective for improving the performance of the model on the task of interest, our experiments show that they do not ensure that the scores assigned to the predictions are good posterior probabilities for the task. Motivated by these results, explore the use of post-hoc calibration (PHC) methods, which are approaches specifically designed to transform scores produced by a classification system into reliable posterior probabilities. In particular, we consider the family of affine logistic regression methods, which is widely used for this purpose and very effective.
We evaluate the systems under comparison using two different metrics. To assess the quality of categorical decisions, we use the standard error rate, which assumes all errors to be equally costly. To assess the quality of posteriors we use proper scoring rules, as recommended by Ferrer & Ramos (025). Tn contrast, recent machine learning papers concerned with this problem frequently use the expected calibration error (ECE) as quality measure (Guo et al. 2017 Si et al. 2022 Ao et al.) 2023). Yet, as discussed by Ferrer & Ramos (2025), the ECF and other calibration metrics do not adequately address the problem of assessing the value of posteriors since calibration only reflects one aspect of their performance (Gneiting & Raftery 2007 Kull
& Flach| 2015 Ferrer & Ramos] 2025). Hence, we instead use the cross-entropy, which, as the expectation of a proper scoring rule, is specifically designed for the problem of measuring the quality of posteriors (Winkler
& Murphy
1968
Gneiting & Raftery 2007 Ferrer & Ramos 2025).
2

---

Published in Transactions on Machine Learning Research (01/2026)
1.1 Summary of our contributions
In this work we propose effective way to combine SFT and PHC for text classification systems based GLMs. The proposed approach, described in Section 5.3 provides the best quality posteriors (as measured by cross-entropy) and, as a consequence, also the best quality decisions (as measured by error rate) compared to using only SFT or PHC. Importantly, our results show that a careful training strategy is required to combine the two methods in order to achieve simultaneously low cross-entropy and error rate. Naively training PHC on the same adaptation data used to train SFT gives the same performance as SFT alone, while splitting the adaptation data for SFT and PHC training is suboptinal. In addition, we show that temperature scaling the most common calibration approach in current machine learning literature-is is often ineffective for calibration of GLM scores. A simple generalization to affine transformation provides significantly better results, sometimes competitive with those obtained with SFT alone. These results are shown in sections 6.2 Finally, in Section 6.6 we show that, after adaptation with the combined approach, the results from a small LLaMA model are similar to those from a significantly larger Qwen model. These results suggest that suboptimal models become competitive after adaptation with our proposed method, though this conclusion will have to be verified by experimenting with a larger number of GLMs. The code needed to replicate the results in this work, along with output scores for all methods under comparison, can be found at
6.3
6.4 and 6.5
2 Previous Works
Many methods for aligning a GLM to a specific task of interest rely purely on Input-Outpin prompting tochniques (Brown et al. 2020). These methods adapt a generic GLM to a given task by carcfully designing
the prompt instructions RRffel et al., 2020), selecting examples that illustrate how to solve the task (Brown
et al. 2020) or retrieving additional information to provide necessary context (Khattab et. al. 2023). More
elaborate methods, such as Chain-of-Thhuuhtt (CoT) (Wei et al. 2022),, Tree of Thoughts (Yao et al. 2023)
or BetterTogether (Soylu et al. 2024), optimize the prompt by iteratively generating and refining context to improve lask performance. Notably, prompting techniques are limited by the finite context length and increase runtime during deployment with respect to simple prompts. In this work, we focus on simple prompts for clarity and efficiency, while noting that our proposed method is compatible with and can be layered on top --ore elaborate prompting strategies such as the ones mentioned above.
SFT methods can adapt single model to one or multiple tasks using in-domain adaptation data Wei et al. )021). Given the large size of modern GLMs, fine-tuning all of their parameters requires a substantial amount of data and computational resources (Chung et al.) 2024). To mitigate this problem, parameter- efficient fine-tuning (PEFT, Lialin et al. 2024) techniques have been proposed in the last few years, including additive (Lester et al. 2021 Houlsby ct al. 2019), selective (Donahue et al., 2014 Gheini et al. 2021), ) and reparamelrization (A (Aghajanya et al. 2021 Liu el al.] 2024) variants. In particular, LoRA (Edward Hu let al. 2022), a reparametrization method, has shown exceptional performance across numerous tasks. LoRA roduces the number of trainable parameters by replacing the lincar operations inside the network with low-rank approximation based on the hypothesis that the difference between the new and the old parameters lies in a low-rank space. Importantly, the adaptation cost of SFT methods is restricted to the fine-tuning stage. The resulting model can run faster than those that use prompt-engineering because the prompts do not need to describe the task. On the other hand, even when using PEFT methods, adapting a GLM requires more data than what is needed for prompting (Mosbach et al., 2023 Zhang et al. 2023 Edward J Hu et al. 2022). We will use a LoRA-basod SFT method which provides strong empirical performance and favorable balance between training cost and adaplation effectiveness.
For the specific case of text classification tasks, another family of adaptation approaches consists of intro- ducing a PHC transformation to convert the posteriors generated by the model into new posteriors that are better matched to the task of interest. For example, Zhao et al. (2021) proposed transforming the posterior for each class by dividing it by the prior for that class, estimated by running the GLM with a content-free query. Similarly, Fei et al. (2023) propose dividing the posterior for each class by the average posterior for a query consisting of in-domain random words. These works implicitly assume that the prior class distribution for the task of interest is uniform. Since this assumption holds approximately or exactly for many text
3

---

Published in Transactions on Machine Learning Research (01/2026)
classification datasets, the methods work quite well for those cases. In our prior work (Estienne et al. 2028
we formalized the problem of bias mitigation and showed that superior results can be obtained compared to the approach proposed by Zhao et al. (2021) by shifting the log posteriors with a bias term trained by mini- mizing the cross-entropy on adaptation data. Further, we showed that the bias term can also be succossfully estimated without annotated data, relying only on knowledge of the class priors for the task of interest. In a recent. work on PHC for NLP, SSpiess et al. (2024) use a Platt scaling calibration trained on labeled data to determine whether a code generated by a GLM is correct or not. In this work, we explore various PHC approaches and combine them with SFT, leveraging their ability to produce better posteriors while incurring negligible additional computational cost during inference.
A few works on text classification with LLMs focus on the problem of producing good quality posteriors, in addition to the categorical decisions. For instance, Li et al. (2025b) propose to mitigate the bias of the model due to each part of the prompt - hypothesis and premise- by normalizing the sequence posterior with the one obtained by prompting the model with each of those parts separately. Alternatively, [Jiang et al., (2023) produce class posteriors by averaging the sequence posteriors over an ensemble of prompls created minor modifications of the original prompt. In other cases, the posteriors or confidences are generated by explicitly asking the LLM to verbalize these probabilitics (Tian et al. 2023 Xiong ct al.) 2024 or by sampling the model multiple times and then counting the frequency of each class (Gal & Ghahramani 2016
Wei & Zou 2019). All those methods are unsupervised, not requiring any data for the task. When some in-domain data is available, recent evidence indicates that supervised methods like SFT or PHC can result in better performance. For instance, in our previous work (Estienne al., 2023). we showed that PHC models trained with a small amount of in-domain data outperformed unsupervised methods. Additionally, Li et al.) (2025a) show that fine-tuning- based approaches perform systematically better than in-context lcarning-basct approaches. Finally, a PHC method proposed by Shen et al. (2024) was also shown to produce higher-quality outputs in comparison with unsupervised methods. For this reason, in this work we focus our efforts on supervised approaches.
The metrics used to assess quality of the text classification systems are usually focused on the quality of
the categorical decisions and include accuracy, error rate, Fl-score or precision/recal (Yin et al.] 2019 Schopf ct al. 2023 Gera et al. 2022 Gretz et al., 2023). Works that are interested in the quality of the posteriors usually report performance in terms of calibration error (Desai & Durrett 2020 Kapoor et al. 2024 Stengel-Fiskin et al. 2024) which, as mentioned above, is a problematic practice since these metrics do not adequately reflect the quality of the posteriors. Very few works on text classification report performance in terms of PSRs and it is usually done in addition to ECE (Xic et al. 2024 Tian et al., 2023), often loading to ambiguous conclusions. For example, in Tables 2, 3, and 5 of the work done by Tian el al. (2023) we can see that the Brier scores (which is the expectation of a PSR) for the TriviaQA datasct are best for the baseline posteriors, but the ECE is best for the proposed calibration methods. This happens because the calibration methods degrade discrimination with the unintended effect that the overall quality of the posteriors is degraded. Faced with the need to select one of the two systems, one should choose the one with the lowest Brier score, not the one with the lowest ECE since those system's 's posteriors are poorer. Therefore, in this work we assess the quality of the posteriors using proper scoring rules (PSRs). For completeness, we include results for ECE and another calibration metric called relative calibration loss (Ferrer & Ramos, 2025) in Appendix C.1
3 Computing text classification posteriors using GLMs
There are many ways to use GLMs for downstream tasks. The most common way is to simply prompt them with a question or instruction and have them generate an answer. In some scenarios, though, this approach may produce answers that do not satisfy the roquirements of the task. One such case is text classification, where the goal is to annolate a certain text with a class label selected among a pre-defined set of possible classes. A GLM used as a free-text generator may not necessarily respond with one of the valid classes for the task. Further, with this approach, the GLM does not produce posterior probabilities for the classes which, as we have argued, may be requirement for high-risk tasks. In this section, we describe a general and principled way to use GLMs for producing class posteriors for text classification tasks.
4

---

Published in Transactions on Machine Learning Research (01/2026)
Prompt p(x, Ua)
diin_atare |sayttn
Deternine the category of the article given by the ueer.<lin_end
<lin_start /uuser
Prompt template
The Blair-Caldvell African Anerican Research Library is branch of the Denver Public Library in Denver, Colorado, in the United States that servea the Five Pointa neighborhood. [..r] </an_end]>>
Sample x
Class identifier
<lin_start start|>assistann Educational Inatitution
Figure 1: Example of sequence p(x, y2) for the Qwen2.5 model for DBPedia sample, where is the sample text taken from the dataset, and y2 is the label corresponding to class 2 of the DBPedia dataset. The token sequence used to compute the posteriors, a2 is the part that identifies the class and is highlighted in dark purple ("Educational Institution," in this example).
The first step is to determine an appropriate token sequence with which to query the GLM. In this work, we use the procedure formalized by Liu et al. (2023), where a model- and task-dependent template p(-) is used to create the sequence of tokens that make up the full sentence (a prompt followed by the answer). That is, given an input text x to be classified and class ys E y A "i......k], we create token sequence, (p. ox) A p(x,2k) yk) which concatenation of the prompt p, consisting of the input lext and some additional a (M&)), representing name for the class yı (see Figure [ instructions, and the answer aę (a,(1)..... ag for an example). Details on the token sequence used for each dataset and each model can be found in Appendix A
Given a sample text x we can then create a prompt p(x, Mee) for each possible class, yk. and run the GLM to obtain the posteriors for each token ax(m) in the answer conditioned on the previous tokens, (ax(aa(m) ag (m 1.......0 .1),....((().pp))) Then, we can compute the posterior for the answer given the prompt as
My
PLM(aR (p) = II (tx((a(m) am(mm 1.,..................... (0),))
(1)
where an (0) is empty sequence. Finally, can define task probability Prast x) for the class label Us E ) given the input x as a normalized version of PaM(as p) that sumn to one over all the valid classes:
PLM(ax 1p) E/= Pam
Prisk (ykk x) A
(2)
This is natural way of using the score produced by the model to obtain probability for each class. In particular, if each class is represented by a single token, Equation 2 is equivalent to applying the softmax operator to the vector obtained by stacking the logits of the tokens corresponding to each class. However, Equation
2 contemplates the general case where the classes may be represented by a sequence more than one token.
This simple approach allows to work around the problem posed by free-text generation, whereby two different sequences of tokens may correspond to the same class. For example, for the polarity classifica- tion task, when prompted with the instruction 'Determine if the following review is positive negative', the GLM may respond 'The review is negative' or just 'Negative'. These two answers would result in different posteriors. We could atlempt to force the GLM to produce only the class name in the answer by adding instructions in the prompt. Yet, this would result long prompts, specially when there are a large number of classes or when each class has a long description, and would not fully guarantee that the GLM will comply. Instead, we simply query the GLM with the aş sequence, with the desired answer included and the token posterions of the answer to compute Equation 2
5

---

Published in Transactions on Machine Learning Research (01/2026)
4 Proper scoring rules for assessment of classification systems
In this work, we follow the literature on strict proper scoring rules (SPSRs), which were proposed decades ago for the assessment of the quality of posterior distributions (Winkler & Murphy 1968 Gneiting & Raftery, 2007: Bröcker 2009). The proper scoring rule (PSR) concept. is founded on the assumption that rational decisions (categorical or otherwise) should be made by minimizing the expectation of a cost function of interest. The cost function assigns a numeric value to every possible decision given the true class of the sample. Given posterior distribution over the classes for a given sample, the optimal decision is the one that minimizes the expectation of the cost function with respect to that distriiution (Duda et al.] 2001) The cost of this optimal decision, called Bayes decision, defines PSR (Dawid & Musio 2014] Brummer 2010). In other words, a PSR is the cost of the optimal decision for the given posterior distribution.
PSRs satisfy an essential property: their expectation with respect to a given reference distribution mini- mizod when the posterior under evaluation coincides with this reference. When the minimum is only attained for the reference distribution, the scoring rule is called strictly proper. Strietly proper scoring rules (SPSRs) encourage the distribution under evaluation to be close to the reference distribution across the full simplex. For further details and formal and conceptual explanations on PSRs, we refer the reader to the work by Dawid & Musio 2014)) Brummer (2100), and Ferrer & Ramos 2025 among many others.
Non-strict PSRs can be constructed by selecting a cost function for categorical decisions, making Bayes decisions for that cost function, and then measuring the cost. The most widely used non-strict PSR is the error rate of decisions made with the argmax rule, i.e., by choosing the class with the largest posterior.
Interestingly, strict PSRs can be constructed by integrating over a family of non-strict PSRs with non-zero weights across the full probability simplex (Gneiting & Raftery 2007
Brummer 2010), allowing for an intuitive understanding of the strictness property (Ferrer & Ramos 2025 Different SPSRs can be obtained
by changing the relative weight of each individual cost function within that integral. One example of an
SPSR is the Brier loss, for which the expectation over the data is called Brier score (BS) (Brier 1950 a
commonly used metric in some medical applications (Huang et al. 2020 Van Hoorde el al. 2015). Another SPSR is the negative log-loss (NLL). The cross-entropy (CE) is the expectation of the NLL, a metric widely used as an objective function for training deep neural network models, including GLMs (Wei et al.) 2021;
Raffel et al. former penalizes extremely wrong posteriors more heavily. While Brier loss has a maximum penalty of 1, the NLL can be infinite when a posterior of 0 is assigned to the correct class. We believe this is a desirable characteristic for high-stakes applications where errors may have extreme consequences. For completeness, results for BS are also included in Appendix C.T
2020 Chung el al. 2024p. In this work, we will mostly use CE instead of BS because the
While the CE is perhaps the most widely used loss for model training, it is much more rarely used as an evaluation metric. This may be partly due to the fact that its values are hard to interpret for being unbounded. Yet, the CE can be normalized to make it easily interpretable. Assuming an evaluation dataset with N samples, ((x1), ()))) 1= and posteriors q = "ul"",..,,"" ((8) for each of those samples, with of A Prs x) Vk 1,..., K, the normalized CE (NCF. for short) is given by:
,...,
EN EK ((v))=m)) log 4
(0)
NCE
(3)
Σk No. (Ng(/N
where N₂ is the number of times the class y/ appeared in the evaluation set Ferrer 2024 The normalization is done by dividing by the CE of the best naive system, one that always outputs the prior distribution of the classes, having no access to the input sample. This CE turns out to be the entropy of the prior distribution. Systems with NCE values close to 1.0 have performance comparable to that of the best naive system.
In this work, as is usually done in machinc learning literature, categorical decisions are made by selecting the class with the highest posterior probability. These decisions, commonly referred to as argmax decisions, can be shown to optimize the error rate, ER (or, equivalently, the accuracy which is given by one minus the error rate). The ER assumes that all incorrect decisions have the same cost. As mentioned above, when decisions are made with the argmax rule, the resulting ER is also an expected PSR, although not a strict one. As with the CE, the E. can be normalized by the ER of system that makes always the same decision
6

---

Published in Transactions on Machine Learning Research (01/2026)
based solely the class priors, without access the input sample. The normalized error rate (NER) of argmax decisions is given by:
Ei=1 N 1 fargmax, "2 Zy(")
NER
(4)
Σk (argmax, (argmax,NN Nadly())
Importantly, although both metrics are PSRs, low NER. does not necessarily imply low NCE. As we will show in the experimental section, while the system's outputs may be good for making argmax decisions, resulting in low NER, they may still be poor posterior probabilities of the classes given the input, resulting in a high NCE. Conversely, a lower NCE does not directly imply a lower NER, though, in practice, this is very often the case. For these reasons, it important to report both metrics when the quality of the posteriors is of interest, in addition to the quality of the decisions.
5 Adapting GLMs to produce good-quality posteriors
In this section describe the methods explored to adapt a GLM to text classification task. We divide the methods into three categories: (1) supervised fine-tuning (SFT), (2) post-hoc calibration (PHC) and (3) combinations of the two approaches.
5.1 Supervised fine-tuning (SFT)
In this work, the LoRA method is implemented for fine-tuning (Edward J Hu et al. 2022) using dataset of sequences created as described in Section 3 from the adaptation samples for the task of interest. We believe the main conclusions from this work should hold for alternative variants of LoRA and other SFT approaches. We leave this exploration for future work.
The optimization loss for SFT is given by the token-level CE corresponding only to the tokens of the answer. That is, given an adaptation set {p((1) (( ) where each sample i consist of sequence of tokens p()) (the input prompt) and sequence of tokens (1)) E {(a,...,ax}) (the desired answer), the training loss is computed as
N
M
(0) ΣM
Σ
log PL. (a (m) (m - 1),..., (0) (0),
(5)
Mi
i=1 m=1
where M, is the number of tokens in answer (t)) 0 is the set of trainable parameters, and a a0 is empty sequence. We also considered computing the loss over the tokens for the full sentence, the prompt and the answer. We found both versions to give similar results, with the one computed only over the answer being faster to compute. For that reason, all experiments in this work are done with the loss computed only over the answer.
When training or fine-tuning large models, is often effective to use regularization. This prevents overfitting, which is particularly important when the amount of training data is small, in scenario of interest. The LoRA approach to fine-tuning can be seen as regularization approach since it greatly reduces the amount of trainable parameters with respect to full fine-tuning (Biderman et al.) 2024). Yet, given that our results showed that models fine-tuned with LoRA still suffer from overfitting, we compared the following approaches:
SFT: Model fine-tuned with LoRA and no further regularization, trained until convergence of the training loss using the full adaptation set.
SFT-ES: Model trained with LoRA by leaving 30% of the adaptation set for validation, and applying early stopping (ES) based on the CE loss on this set.
rer-ES--e Model trained on the full adaptation set for a mumber steps equal to the optimal number obtained for the SFT-ES model. This approach requires two fino-tuning steps, one to obtain the optimal number of steps using 70% of the data for adaptation and one to obtain the final model using 100% of the adaptation data.
7

---

Published in Transactions on Machine Learning Research (01/2026)
SFT-L2: Model fine-tuned the SFT one above but with training loss given by (1 (1=)))
where X E [0. 1] and |1 is regularization term given by the 1.2 norm of the model's parameters.
SFT-LS: Same as SFT-L2 but using a label smoothing regularization term in (Meister et al. 2020 instead of an L2 term. The loss is then given by (1 - (0) + XC where À is again a fixed scalar between O and and (f) corresponds to the Kullback-Leible distance between the posteriors distribution over tokens produced by the GLM and the uniform distribution.
Implementation details for each method can be found in Appendix B After the GLM is fine-tuned, the class posteriors are computed using Equation (2), as for the unadapted system.
As we will see in the results, the best performance in terms of NER is achieved without further regularization beyond that provided by the LoRA approach, while regularization methods like label smoothing carly stopping result. in the best NCF. This poses apparent dilemma. Which approach should we choose if we care both about having good categorical decisions and also good posteriors? Fortunately, as we will see, we do not need to settle for onc or the othor, since a combination of SFT and PHC leads to the best performance on both metrics.
5.2 Post-hoc calibration (PHC)
PHC approaches consist of adding a final stage to a classification system to transform its oulputs, q, into better class posteriors, q (Silva Filho et al.] 2023 Ferrer & Ramos 2025). In our case, the system's outputs are given by Equation 2 computed with token posteriors obtained from the unadapted or the fine-tuned GLM. One of the simplest calibration methods involves applying affine transformation to the logarithm of the posteriors of the system, followed by softmax transform to obtain a new set of posteriors:
q = softmax A log(g) + B),
(6)
where q, q, and B are vectors of dimension K, the number of classes, and A E R If the parameters A and B of the affine transformation are trained to minimize the cross-cntropy, the resulting model will produce the best possible posteriors on the adaptation data. If the transformation does not overfit that data, then those posteriors will also be good on unseen data.
This transform was proposed by Kull et al. (2019) under the name Dirichlet calibration, and under the name matrix, vector and temperature scaling, depending on the form of A and the restrictions on B, by (Guo all, 2017). For the binary case, this transform is known Platt scaling (Platt 2000). The case where A is scalar was probably first proposed by (Brummer & Van Leeuwen (DP) calibration
2006 under the name direction-preservingg
In our experiments, we will use four version of this transformation:
Vector Scaling (PHC-VS): Matrix A assumed to be diagonal.
Direction Preserving (PHC-DP): Matrix A is replaced by a scalar, a.
Bias Only (PHC-BO): Same as PHC-DP, but with the weight a fixed at 1.
Temperature Scaling (PHC-TS): Same PHC-DP but with the bias term fixed at O.
In all cases, the parameters are trained to minimize the cross-entropy of the training data. We also explore adding a regularization term, as for the SFT methods. In each step, the value of the loss is computed using all calibration samples, and if the value did not decrease for 10 steps, the training is stopped.
In addition to the family of affine PHC methods, we also evaluate the use of Adaptive Temperature ure Scaling (PHC-Ada-TS), proposed by Joy et al.) (2023). This approach uses the embedding from the last layer as input to a variational autoencoder (VAE) model that produces the temperature for calibrating each sample.
As will see in the results, PHC is less effective than SFT in terms of NER for all adaptation sizes. It is also inferior in terms of NEC for the larger adaptation sets. Yet, when the adaptation set is smaller, PHC
Some of the methods mentioned take logits instcad of log posteriors as input. Yet, it can be casily seen that both expressions are identical by showing that, if are the logits, then q softmax and softmax( (A log(q) + (8) softmax (Az + 9), since log(q) + and the outer softmax transform normalizes out the shift. c.
8

---

Published in Transactions on Machine Learning Research (01/2026)
approaches can, in some cases, result in better NCE than SFT. Fortunately, the combined method described below leads to the best performance on both metrics, consistently outperforming matching the best of both methods on all tested scenarios.
5.3 Combining PHC and SFT
SFT can lead to overfitting, even when using LoRA, causing the model to produce suboptimal (overconfidennt) class posteriors. This overfitting, though, can be solved by applying PHC on the posteriors produced by the model after SFT. However, combining these two approaches is not trivial. The naive approach would be to use the same adaptation samples to fine-tune the model and train the calibrator. This approach does not work since models trained with CE are already well-calibrated on the samples used for training. Hence, training a calibration model using the posteriors produced by the fine-tuned model for the samples used for fine-tuning would lead to a calibration transform close to the identity function. What we wish to fix is the miscalibration that occurs on samples unseen during fine-tuning, i.e,, samples with a score distribution that resembles the one we will see during deployment. To this end, ideally, we need two different adaptation sets, one for SFT, and one for PHC. Yet, given a small amount of adaptation data, splitting the data. to train the two models is quite suboptimal.
To make the best possible use of the limited amount of adaptation data, we propose the following procedure for combining SFT and PHC:
1. Perform SFT on 70% of the adaptation set, holding out 30% of the samples for the next step, but training until convergence of the training loss.
2. Train the parameters of the calibrator on the samples held out from fine-tuning, using the posteriors produced on that data by the model obtained above. These posteriors are unbiased since those samples were unseen for that model.
3. Perform LoRA-based SFT using all the adaptation samples until convergence. This is the model called "SFT" in Section 5.1
4. Apply calibration to the predictions of the SFT model from step 3, using the parameters from step 2.
We will refer to this procedure as "SFT + PHC". Note that this method can be implemented with all the PHC variants described the previous section.
6 Experiments
We conducted experiments using the instruction-tuned versions of LLaMA3.2 (Grattafiori et al. 2024 with 1 billion parameter&]] and Qwen2.5 (Yang et al. 2024), with 7 billion parameter five different classification tasks. The two models were selected to be relatively small to allow for a large number of experiments, while being different in terms of size and performance, with the Qwen model being seven times larger and significantly botter than the LlaMA model. The fact that, as we will see, the main conclusions of this work are similar for these two models suggests that the benefit of the proposed approach is not dependent on the particular GLM design choices or its base performance.
In the next sections, first describe the experimental setup and compare the results using LLaMA3.2 for the various adaptation approaches considered in this work. Then, explore in detail the impact of the size of the adaptation set on subset of selected approaches. Finally, we compare the results for the two GLMs.
6.1 Experimental set-up
We focus on downstream tasks that can be framed as standard text classification problems, including results for the following five open-access datasets:
hhtte///wwgginiccc...c/ttttaaaaa http:///aggingfacc.co///e/// Instrruct
Instruct

---

Published in Transactions on Machine Learning Research (01/2026)
SST2 (Socher et al. 2013) the Stanford Sentiment Treebank, which includes movie reviews annotated as either positive or negative. We used the GLUE version of the dataset (Wang et al. 2018), which does not contain publicly available labels for the test split, so we used only the training and validation sets. We used 400 samples for ovaluation selected from the combination of the original validaaan sets to ensure consistency of the label frequency between training and evaluation sets.
AGNews (Zhang et al. 2015]: a collection of news articles grouped into four categories. Samples were downloaded from the fancyzhx/ag_a aanes huggingface repository, which contains the original version of the dataset. We used 400 samples for evaluation selected after pooling the original train and test sets.
DBPedia (Lehmann et al., 2015): a dataset derived from Wikipedia, where each article is categorized into one of 14 topics. Samples were downloaded from the fancyzhx/ /bpedia 14 huggingface repository, which contains the standard 14-category version of the datasct. We used 700 samples for evaluation sampled from the original test set.
20NewsGroups Lang 1995]: posts from online newsgroups categorized into 20 different topics. Sam- ples were downloaded from the SetFit/20 _ huggingface repository, which contains a curated variant of the original dataset. We used 800 samples for evaluation selected after pooling the original train and test sets.
Banking77 (Casanueva et al. 2020): customer service queries related to online banking, classified into 77 intent categories. Samples were downloaded from the 20lyAI/banking77 huggingface repository, which contains the original version of the datasct. We used 1000 samples for evaluation selected after pooling
the original train and test sets.
Appendix A jinlludes details of the prompt template, the class prior distribution, and the number of evaluation samples for each dataset.
We consider the scenario where a small amount of in-domain samples available for adapting the model to the downstream tasks and study the impact of the adaptation set size in the performance of the different approaches. To this end, we generate adaptation datasets of varying sizes, randomly selecting N samples from the subset not used for evaluation. The value of N was computed as N = K 2", i.e., proportional to number of classes, K, with the factor being power of 2. We initially tried using the same set of factors T for all (asks. Yel, we found that the same T resulted in very different trends for tasks with two classes versus tasks with many classes. In particular, while = 1 was too small for SST-2 and AGNews, with 4 and 8 samples being insufficient for adaptation, 154 samples were enough to obtaín reasonable results for Banking77. We then concluded that in order to obtain similar trends across datascts we needed to make T depend the number of classes. Parameterizing T as T'/log(K) and then rounding the result. to the nearest power of 2, resulted in similar trends across datasets for the same value of T" E {4,5, 6, 7, 8). For the SFT methods that require a validation set, after selecting N adaptation samples, we further split those N samples, keeping 70% for fine-tuning and the rest for validation.
6.2 SFT-based adaptation
Figure 2 shows the results for the LLaMA3.2 model after adapting it to each of the datasets above using the SFT methods described in Section 5.1 Each plot shows two groups of bars, corresponding to the training = sizes obtained with T' = 4 and T 8. Each bar shows the median NCE or NER across S adapted models obtained by sampling the adaptation data with different random seeds. We used S = 5 seeds for DBPedia, 20Nowsgroups and Banking77, for which the results were stable across scods, and AGNews, for which the results were noisier.
sceds for SST-2 and
The figure shows that the categorical decisions obtained with the unadapted model (NoA) perform better than chance, as indicated by NER values lower than 1.0. On the other hand, the posteriors produced by the unadapted model are extremely poor with NCE values larger than 1.0 for all datasets except. SST-2. As expected, SFT provides large gains in terms of both metrics in most datasets even for the smaller adaptation set, though, naturally, the gains are larger for the larger adaptation set. Note that, in these results, SFT is always performed using LoRA, which provides strong regularization compared to full fine-tuning (Biderman) et al. 2024). Tn fact, in our preliminary experiments full fine-tuning produced extremely poor results for
10

---

Published in Transactions on Machine Learning Research (01/2026)
SST-2
AGNeWS
DBPedia
20 Newsgroups
Banking77
NoA 1.49
2.01
Nok.
0.4-
0.2
NoA 0.60
Nok 0.61
NoA 0,88
0.4
NER
0.2
32
512
N
512
N
896
80
1280
154
2464
Adaptation sizes
Methad
NoA
SFT-L2 0.100 SFT-L2 (A 0.011
SFT-ES SFT-ES-retrained
SFT-LS (A 0.500
SFT-LS (A 0.00))
SFT
SFT-LS 0.110)
Figure NCE and NER for the unadapted model (NoA) and LoRA-based SFT-adapted models for two adaptation samples (setting T' and T 8) on TLaMA3.2 model. Bars divided by groups of colors according to the regularization method used: red for L2 regularization (SFT-L2), blue for early stopping (SFT-ES), green for label smoothing (SFT-LS) and gray for no additional regularization beyond that provided by the LoRA approach. The bar height corresponds to the median across seeds used to select adaptation samples, and confidence intervals (black vertical lines on top of the bars) correspond to the 1st and 3rd quartiles. The NoA values are printed in the top right corner when they fall outside of the y-range.
the smaller amount of adaptation data and no advantages over LoRA-based SFT for the larger adaptation sizes. For this reason, full fine-tuning results are not shown in this paper.
Comparing the baseline LoRA-based SFT results with those that include an additional regularization ap- proach like L2, LS or ES, we can see that the additional regularization provides no benefit in terms of NER, often causing slight degradation in that metric. In contrast, some regularization approaches namely, WET-ES-reetrin and SFT-LS consistently improve the NCE metric. This indicates that LoRA-based SFT still suffers from some degree of overfitting, which affects the quality of the posteriors. The additional reg- ularization provided by the early stopping and the label smoothing methods mitigates this overfitting and, as consequence, improves the quality of the posteriors which is reflected in a lower NCE.
Comparing SFT-ES with SFT-ES-retrained terms of NCE, we can see that the retraining step on all the available adaptation data after selecting the number of steps by holding out some validation data (Section 5.1) gives significant gains. For the SFT-LS approach, datasets and training data sizes which means that this parameter would have to be tuned for every new usc-case scenario. This would require approach similar to the one used for choosing the number of step for SFT-ES, leaving some validation data to select the À and then retraining on all data using the best value. This would be extremely costly since it would require running SFT-LS several times one for each value of X be explored and final one on the complete adaptation set. Fortunately, as we will see in Section 6.4 the miscalibration produced by SFT can be addressed by applying a post-hoc calibration stage, a more effective and efficient alternative to the regularization approaches compared in this section.
that single value of À is optimal across all
11

---

Published in Transactions on Machine Learning Research (01/2026)
SST-2
AGNews
DBPedia
20 Newsgroups
Banking
1.4
1.49
2.02
Nok 2.24
1.2
1.0
0.8
0.6
0.0
0.8
NER
N 32
512
N 32
N 512
N 896
N
N 1280
N 154
N = 2464
Adaptation sizes
Method
NoA
PHC-TS-LS (M= 0.500
PHC-DP-LS ( =
PHC-VS-LS 00.5)
SFT
PHC-TS-LS (A
PHC-OP-LS (A=
PHC-VS-LS 0.11
SFT-ES-retrained
PHC-TS
PHC-DP
PHC-VS
PdC-Ads-TS
Figure 3: NCE and NER for the unadapted model (NoA) and PHC-adapted models for two adaptation samples (setting T" 4 and T' 8) LLaMA3.2 model. The type of transform is identified by the color of the bars, while the intensity identifies the degree of label smoothing regularization. We also include results for SFT and SFF--ES-retrained from Figure 2 for reference. The bar height corresponds to the median across seeds used to select adaptation samples, and confidence intervals (black vertical lines on top the bars) correspond to the 1st and 3rd quartiles. The NoA values are printed the top right corner when they fall outside of the y-range.
6.3 PHC adaptation
Figure g shows the results for the PHC methods explained in Section 5.2 The SFT and SFT-ES-retrain results from Figure 2 are also shown for reforence. We can see that TS, the most common PHC method in current machine learning literature (Guo al. 2017), and Ada-TS are significantly worse than DP and VS calibration, both of which include bias term in addition to a scaling factor. Bias-only (BO) calibration is excluded from the Figure reduce clutter but results from this method were always between TS and DP results. As explained by Ferrer & Ramos (2025), the bias term in the calibration transform compensates for mismatches in the prior class distribution between the model and the task of interest, while the scale compensates for overfitting or underfitting problems. The results in Figure 3 suggest that a large source of miscalibration for this GLM is prior mismatch, since the bias term provides significant gains over the scale-only method.
The figure also shows the impact of label smoothing (LS) regularization. This approach was selected because it was superior the other two regularization approaches tried: L2 regularization and early stopping. We that regularization provides no benefit for TS and DP calibration. On the other hand, for VS, which contains much larger number of parameters than DP and TS, some degree of regularization is beneficial, particularly for the smaller adaptation sets.
Overall, we can see that PHC-DP and PHC-VS with À = 0.1 provide large gains with respect to the unadapted model, but are still generally worse than SFT-adapted models with two exceptions: for the smaller adaptation sizes for 20Newsgroup and Banking77, PHC-DP and PHC-VS provide similar results SFT.
12

---

Published in Transactions on Machine Learning Research (01/2020)
SST-2
AGNewS
DBPedia
20 Newsgroups
Banking
1.49
2.02
Nok.
1.2
1.0
0.8
0.6
0.2
0.0
0.8
0.6
NER
32
512
512
N
N 896
N
N 1280
154
= 2464
Adaptation sizes
Method
SFT + PHC DP 9 0.51
SFT + PHC TS (.55
SFT
SFT P5
SFT + PHC DP 0 0.11
SFT PHC 0.1)
SFT-ES retrained
NoA
Figure 4: NCE and NER for combined adaptation methods for two adaptation samples (setting T 4 and T' 8) I.aMA3.2 model. The color is determined by the type of PHC approach while the intensity depends on the degree of regularization. We include the SFT and SFT-ES-retrahh results reference since they are the overall best performing approaches for NER and NCE, respectively, from Figure 2 The bar height corresponds to the median across seeds used to select adaptation samples, and confidence intervals (black vertical lines on top of the bars) correspond to the 1st and 3rd quartiles. The NoA values are printed in the lop right corner when they fall outside of the y-range.
6.4 Combined SFT+PHC adaptation
As saw in Section 6.2 the best categorical decisions measured by the NER are obtained with LoRA- based SFT without additional regularization. On the other hand, the best posteriors as measured by NCE are obtained by further regularizing the fine-tuning process, indicating that LoRA-based SFT still suffers from some degree of overfitting. While the regularization approaches explored in Section 6.2 improve over the SFT baseline terms of NCF, they require tuning of the regularization parameter, very costly endeavor. In this section, we explore the use of PHC as an alternative to regularization improve the quality of the posteriors obtained with SFT.
Figure 4 shows the performance of adaptation methods that use SFT in combination with PHC using the proposed strategy from Section 5.3 and exploring the use of label smoothing when training the PHC model as Figure 3 We apply PHC to the model obtained with SET without additional regularization. We do not show results combining PHC with further regularized versions of SFT because their performance is comparable to or worse than applying PHC SFT dircctly. PHC is better compensating for the overfitting produced during LoRA-based fine-tuning than those regularization approaches, and their combination does not provide any further gain. Finally, adaptive TS is not included in this figure because it gives similar worse results than standard TS.
Comparing results across the three PHC approaches, we can see that PHC-TS consistently produces the best results for all datasets. This happens in contrast to the results for PHC-only, where a more complex transform like DP or VS was needed. The fact that, after SFT, the best PHC approach one without bias term suggests that the mismatch in the priors fixed by SFT and the only problem left for PHC fix the overconfidence in the posteriors produced by overfitting Importantly, the combined SFT + PHC-TS system
13

---

Published in Transactions on Machine Learning Research (01/2026)
SST-2
AGNews
DBPedia
20 Newsgroups
Banking
NoA
2.02
NoA 1.09
NoA 2.24
1.04
0.9
0.5
0.3-
0.45
0.25
0.7
0.5
0.4
0.5
0.3
0.1
0.4
0.25
0.05
0.4
0.2
112
0.55
NiA
0.58
NoA
0.88
0.1
0.55
0.5
0.45
0.35
0.25
0.5
0.4
0.4
0.3
0.25
0.34
0.2
0.1
0.35
0.2
0.15
32
512
250
112
154
Number of train samples
Method
NoA
SFT-ES-retrained
SFT
PHC-DP
SFT PHC-TS
Figure 5: Performance of selected adaptation methods versus the number of adaptation samples on LLaMA3.2. Note that the NER curve for SFT is behind the one for SFT PHC-TS, since TS does not change the NER value.
is consistently better than or comparable the best SFTT-nll approach in terms of NCE (SFT-ES-retrain) while being comparable the best SFTT-nnly approach in terms of NER (SFT).
Analyzing the impact of regularization, can see that, for the TS transform, LS helps only in one scenario: SST-2 with the smallest dataset. Notably, regularization helps DP and VS but it does not make them better than TS, strongly supporting the hypothesis that the only aspect to be fixed after SFT is the overconficlncce of the posteriors produced by overfitting. This problem can be successfully compensated by a single scale in the log-posterior domain.
In summary, LoRA-based SFT without further regularization followed by PHC-TS consistently gives the best results for both NCE and NER across all the evaluated adaptation approaches, eliminating the need for trade-off between the two metrics. Further, in addition to performing better, this procedure is also computationally more efficient than applying further regularization approach during LoRA-based SFT, which requires tuning of the regularization weight, or the early stopping epoch. While regularization of the TS transform provides some benefit in one of the test scenarios, properly tuning the À hyperparameter for regularization would require either an additional development set or a computationallyycon procedure on the available adaptation data. For this reason and given the marginal gain that regularization of PHC-TS provides, for the rest of the experiments use the unregularized version. Yet, we note that, when a very limited amount of data is available, it may be worth exploring the use of regularization, using careful pross-validation approach.
gross-validation
Importantly, the benefit of applying PHC after SFT cannot be obtained if the same data used to train SFT is used to train PHC. This because, when trained on the same data as SFT, the PHC transform is the identity, since calibration on that data is already perfect. These results are not shown in the plot since they coincide exactly with the SFT results. Hence, the two-stage process proposed in Section 5.3 is essential for achieving superior model combining SFT and PHC.
6.5 Detailed impact of the adaptation set's size
Figure 5 shows the performance of the unadapted model (NoA) and selected adaptation methods for different number of adaptation samples, N, obtained by varying T' {4,5, 7, 8). As in Figures 3 and 4
selected
14

---

Published in Transactions on Machine Learning Research (01/2020)
SST-2
AGNeWs
DBPedia
20 Newsgroups
Banking77
12-
KoA (LLama333
N5A 12wan22 781 0.32
2-181
0.6
Now PB) 0:78
(LLane 3.2 180
0.45
NoA 3.Lame3 2-180 2.01
NJA (0wan2.5 5 2.22
3.0
NoA (Liama3 2:180
NoA [0wun2: 5-78) 1.52
2.2
NoA (Llamal 2:18) 2.24
NDA 1Qwon2.5 7 3.62
0.9
1.0
0.5
0.37
0.5
1.0
0.3
0.7
0.2
2.5
0.4
0.3-
0.35
0.4
0.1
0.2
0.05
0.3
0.2
64
12B
256
512
32
64
128
258
512
56
11.2
224
648
896
150
320
640
1260
15.4
3:08
616
12:22
2464
0.7
0.75
0.6
NSA (epen...5 781 0:11
0.45
NoA (Qwan2. 75) 0.27
3-2-00
0.35
NsA (Qwan2.577) 75) 0.30
3.21m
0.65
0.7
NoA (Qwan2 5-78) 0.80
Nok (Cwan2.55 78) 0.85
0.5
0.4
0.3
0.6
0.4
0:35
0.55
0.2
0.5
0.25
0.15
0.2
0.2
0.1
0.4
0.3
0.1-
0.3
0.2
32
64
128
256
512
32
64
128
256
512
56
112
224
448
896
80
160
320
640
1260
0.3
154
3.8
616
1232
2464
Number of train samplles
LLama3.2-18
PHC-DP
SFT
SFT PHC-TS
Qwen2.5-7B
Figure 6: Results for LLaMA3.2 and Qwen2.5 models while varying the number of adaptation samples. Solid lines correspond to the LLaMA model, while dashed lines correspond to the Qwen model.
SFT and SFT-ESS-etrain from the fine-tuning methods, for being the overall best methods of their family for NER and NCE, respectively. We also select PHC-DP and SFT + PHC-TS to represent the calibration-only and combined groups, respectively. The results for the left-most and right-most points in each plot coincide with those for the corresponding methods in Figures 2 3 and 4
Overall, we can see the expected trend of degradation as the number of adaptation samples decreases for all methods, with a sharper degradation for SFT methods than for PHC-DP. When enough samples are available for adaptation, SFT is significantly better than PHC. However, as the number of samples gets smaller, PHC-DP outperforms and Banking77).
in terms of NCE in two of the five datasets (20Newsgroups
The figure shows that conclusions from the previous section are consistent across all intermediate adaptation sizes: while SFFT-ES-retriin is better than SFT for NCE, the trend is reversed for NER. Hence, if we had to select one SFT approach, we would have to prioritize one metric over the other. Further, in some scenarios, PHC-DP gives better NCE than both SFT approaches. Fortunately, the combined approach eliminated the nced for a tradc-off between metrics, reaching the best results across the board. Table in Appendix shows the numeric values corresponding to the methods of Figure 5
6.6 Comparison of results for LLaMA and Qwen models
In order to explore whether trends and conclusions hold across different. GLMs, we run the main experiments using Qwwe2..5-BBIInstrun
model that significantly better than the LLaMA3.2 model used in the previous sections. Figure G shows comparison of results for LLaMA3.2 and Qwen2.5 as a function of the number of adaptation samples. The line style identifies the model, while the methods are represented with the same colors as in Figure 5 Here we only include PHC-DP, SFT and SFT + PHC-TS for clarity.
The figure shows that, for the two lasks with fewer classes (SST-2 and AGNews), unadapted Qwen2.5 out- performs unadapted LLaMA3.2, both in terms of quality of decisions and of quality of posteriors. However, for the tasks with more classes (DBPodia, 20Newsgroups, and Banking77) Qwen2.5 slightly underperforms compared to LLaMA3.2, with both models performing considerably worse than the naive bascline in terms of NCE. After SFT or the combined method, Qwen2.5 perperosms L.L2 consistently, though only by
15

---

Published in Transactions on Machine Learning Research (01/2026)
relatively small margin. These results suggest that, after adaptation, different models converge to similar performance, despite being initially different.
Importantly, as shown in Figure 10 in Appendix C.3 which presents additional results on the Qwen2.5 model, the main qualitative conclusions are the same as for the LLaMA3.2 model. PHC-DP outperforms PHC-TS, in most cases giving large gains with respect to the unadapted model. SFT-ES-retrain outperforms SFT terms of NCE, while SFT has a small advantage in terms of NER. Finally, the main result in this paper holds: SFT followed by PHC-TS gives the best results of all adaptation methods across all combinations of dataset, adaptation size, and metric. These results support the hypothesis that the main conclusions in this work are robust to the choice of GLM.
7 Conclusions
In this work, we developed and compared various approaches for adapting GLMs to downst classification tasks. Our main focus was develop an approach that can produce good-quality class posteriors without sacrificing the quality of the decisions. These posteriors could later be used to make optimal decisions using Bayes decision theory, or simply provided to the user for interpretation. This is particularly important for high-stakes applications, where accepting incorrect decision may have severe consequences.
We evaluated the performance the resulting adapted models in terms of Normalized Error Rate (NER), which measures the quality of categorical decisions, and in terms of Normalized Cross Entropy (NCE), which measures the quality of the class posteriors. We compared simple post-hoc calibration approaches that transform the normalized sequence posterior from the GLM into better class posteriors and supervised fine-tuning using LoRA.
Our results show that the normalized sequence posterior from the unadapted model is an extremely poor class posterior, showing worse performance than a naive system that does not have access to the input text and only knows the class priors. Post-hoc calibration greatly improves the quality of the posteriors and also, as a consequence, the quality the categorical decisions, offering an efficient and effective adaptation approach. While fino-tuning leads to larger improvements than calibration in most cases, it is prone to overfitting, resulting in posteriors that overestimate the model's certainty. To address this issue, we evaluated the use of further regularization approaches in addition to LoRA, showing that label smoothing and early stopping can improve the quality of the posterions, at the cost of requiring an additional tuning stage for the regularization hyperparameter. Finally, we proposed a combined method where a post-hoc calibration model is applied to the output of the fine-tuned model, relying on an ad-hoc algorithm to optimally leverage the limited amount of adaptation data for training both stages. The model resulting from this combined adaptation approach consistently provides the best performance across all datasets for NCE and NER over range of adaptation set sizes. In particular, this combined model outperforms the regularized SFT model, while also being computationally lighter, not requiring tuning stage for regularization parameter.
The proposed combined approach constitutes a straight-forwadd and effective approach for adapting GLM to a specific text classification task using a limited amount of labeled adaptation data, In future work, we plan to explore the integration of prompt engineering techniques for task adaptation, as we believe they could be complementary to the approaches studied this work and further enhance model performance. In addition, we will expand this study to include more complex tasks like factual question answering, and information retrieval, where a posterior for correctness can be obtained with approaches similar to the ones studied in this work.
Broader Impact Statement and Limitations
The methods proposed in this work aim to improve the quality of the class posteriors obtained from generative language model (GLM) for text classification tasks, making their outputs more interpretable and reliable for downstream applications. This has the potential to enhance decision-making in critical areas such as hcaltheare, finance, and legal applications, where incorrect predictions could have grave consequences. By providing better measure of uncertainty, our approach could contribute to safer and more reliable AI systems.
16

---

Published in Transactions on Machine Learning Research (01/2026)
It is important to note that the conchusions of this work are limited to cases where adaptation and evaluation scenarios are matched: the same prompt templates are used in both steps and the samples are extracted from the same domain. Under these assumptions, we believe the proposed adaptation approach should work well across different models and domains. these assumptions do not hold for example, if the prompt is changed to provide examples of the lask during evaluation but not during adaptation the quality of the posteriors for the adapted system likely degrade compared the matched case. More research is required to determine the degree to which model adapted using a given prompt or domain can be used for different prompt or different domain.
Acknowledgments
Research reported in this work was supported by JPMorgan Chase Faculty Research Award and an Amazon Research Award Fall 2023. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views Amazon.
Disclaimer
This paper was prepared for informational purposes in part by the Artificial Intelligence Research group of JPMorgan Chase & Co. and its affiliates (*JP Morgan") and is not product of the Research Department of JP Morgan. JP Morgan makes representation and warranly whatsoever and disclaims all liability, for the completeness, accuracy or reliability of the information contained herein. This document is not intended as investment research or investment advice, or a recommendation, offer or solicitation for the purchase or sale of any security, financial instrument, financial product or service, or to be used in any way for evaluating the merits of participating any Iransaction, and shall not constitute solicitation under any jurisdiction or to any person, if such solicitation under such jurisdiction or such person would be unlawful.
C 2024 JPMorgan Chase & Co. All rights reserved
References
Marah I. Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javahoripi, Picro Kauffmann, James R. Lec, Yin Tat Lec, Yuanzhi Li, Weishung Liu, Caio CT Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, and Yi Zhang. Phi-4 Tochnical Report, December 2024. URL https:/www..microoffft mwcrosoot.....mmmmmmmmmmmocm.... http://www....oo...coo/////aaaaa wwt.///arre///eeeeeee////laaaaa. phi-4-techmiial--repoo
Armen Aghajanyan, Sonal Gupla, and Luke Zeltlemoyer. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine- TuninT In Proceedings of the 59th Annual Meeting of the Association for Compu- tational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 7319-7328, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1 1/2222 lcl-long.568. URL httpp:://aclanhoology
acl-loo 568/
Shuang Ao, Stefan Rueger, and Advaith Siddharthan. Two Sides of Miscalibration: Tdentifying Over and Cnder-Confidence Prediction for Network Calibration. In Proceedings of the Thirty-Ninth Conference on Uncertainty in Artificial Intelligence, pp. 77-87. PMLR, July 2023. URL https://proceedings mlr ISSN: 2640-3498.
press/v216/aosi ao2aa htmm
Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jen- nings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John Patrick Cunningham. LoRA Learns Less and Forgets Less. Transactions on Machine Learning Research, May 2024. ISSN 2835-8856. URL https://openrevieet net./forummii-aaloeenneeee
Christopher M. Bishop. Pattern Recognation and Machine Learning (Information Science and Statistics). Springer, edition, 2007. ISBN 0387310738. URL http://www.amazett com/
Pattern- Restern-Recognttion-Leeeee Swwpgg-Inoooatio-Sttaaatee htt.a//nhh...tttaaaaae////////ta
17

---

Published in Transactions on Machine Learning Research (01/2026)
Glenn W. Brier. Verification of Forccasts Expressed in Terms of Probability. Monthly Weather Review,
January 1950. ISSN 1520-0493. URL https://journalls ametsoc. wttg:viie//ouuraals/mmte////// 2_0_co_2 2. 2m
1520-0493 1950_ 078
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877 www..ii.ss..pppp.ee cc/paper
1901. Curran Associates, Inc., 2020. URL html
Niko Brummer. Measuring, refining and calibrating speaker and languoge information extracted from speech. PhD thesis, University Stellenbosch, 2010. URL https://scholar: sun. ac za/ items/
1b46805b-2ble 46aa 2aaa--83ee-7ddds220115
Niko Brummer and David A. Van Leeuwen. On calibration of language recognition scores. In 2006 IEEE Odyssey - The Speaker and Language Recognition Workshop, pp. 8, June 2006. doi: 10.1109/ODYSSTT URL https:///eeeeppoore ieeeeeeeeeeeeeeeee.eeeoe.eeeeee. hrgg/aastraet//cccuee///1333333
Jochen Bröcker. Reliability, sufficiency, and the decomposition of proper scores. Quarterly Journal of the
Royal Meteorological Sociely, 135(643):1512-1111, July 2009. ISSN 0035-9009, 1477-870X. doi: 10.1002/ qj.456. URL http://mmeettttollnnneeerrrrrrrr wwwp:///......ccooo//oooooooo 1002/qj
lnigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient Intent Detec- tion with Dual Sentence Encoders. In Proceedings of the 2nd Workshop on Natural Language Processing for Conversational AT, pp. 38-45, Online, July 2020. Association for Computational Linguistics. doi: 10. 306533/ww2000.mlp4ooom 1.5. URL
org/2020 nlp4convai-1.5/
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, and et al. Scaling instruetionFFnettnee Language Models. Journal of Machine Learning Research, 25(70):1 53, 2024. ISSN 1533-7928. URL http://jmlr... httgg////orrr/////////////
papers/v25/23 0ttp:///2//33000000010 html
Jeremy Cole, Michael Zhang, Daniel Gillick, Julian Eisenschlos, Bhuwan Dhingra, and Jacob Eisenstein. Selectively Answering Ambiguous Questions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Pro- ceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 530 543, Singa- pore, December 2023. Association for Computational Linguistics. doi:
10t////111222.mm/// mmn..mmmmmmmm-333555 main.35.
URL https::/aclaattholoe hwwphooooooooorg//2000 0223. emnlp-main.38 35/
Alexander Philip Dawid and Monica Musio. Theory and applications of proper scoring rules. METRON, 72 (2):169-183, August 2014. ISSN 2281-695X.
doi: n..1007/s40300-00 11000000000100000 014-0009----00--.. URL bttps://doi.. org/
10. 00.00077443300-014403999
Shrey Desai and Greg Durrett. Calibration of Pro-trained Transformers. In Bonnic Webber, Trevor Cohn, Yu- lan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 295 302, Online, November 2020. Association for Computational Linguistics. doi: 2ttp:////11/2200.mm//mmmamma
URL https:///ccaatthoologg wwwp://y..r......o..//// cannp-maiin 21/
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition. In Proceedings of the 31st International Conference Machine Learning, pp. 647-655. PMLR, January 2014. URL https: html ISSN: 1938-7228.
//proceedings.c weweeediinnsmm.......mm htt.:eeiigg........eeae/////aaaa
Richard O. Duda, Peter F. Hart, and David G. Stork. Pattern Classification. Wiley, 2 edition, 2001.
18

---

Published in Transactions on Machine Learning Research (01/2026)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference Learning Representations, 2022. URL https://openreriex..ne http:///ww...tteooooooooodddttta
Lautaro Estienne, Luciana Ferrer, Matías Vera, and Pablo Piantanida. Unsupervised Calibration through
Prior Adaptation for Text Classification using Large Language Models, August 2023. URL http://arxx
org/aba hog/abs/280 /20p//23070000 2307.06713 arXiv:2307.0611 0067113 [cs].
Yu Fei, Yifan Hou, Zeming Chen, and Antoine Bosselut. Mitigating Label Biases for In-context Learning, In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1:
Long Papers), pp. 14014-14031, Toronto, Canada, July 2023. Association for Computational Linguistics. URL htttpe//aadaatiiooogg.oogg/ wttp://////2....11oooo...... /
doi: 10.18653/vl. 111116533/122233.ar// /022
Luciana Ferrer. No Need for Ad-hoc Substitutes: The Expected Cast is a Principled All-purpose Clas- sification Metric. Transactions on Machine Learning Research, June 2024. ISSN 2835-8856. URL
https://openreviiin http:://pparrreeee..eeeoouuuaada meet/forunidd-SPPrCEE..
Luciana Ferrer and Daniel Ramos. Evaluating Posterior Probabilities: Decision Theory Proper Scoring Rules, and Calibration. Transactions on Machine Learning Research, August 2025. ISSN 2835-8856. URL
https://openrevieet http:p/wwww..eeeeeeeommtttetttpp
Yarin Gal and Zoubin Ghahramani. Dropout as Bayesian Approximation: Representing Model Uncertainly in Deep Learning. In Proceedings of The 33rd International Conference on Machine Learning, pp. 1050- 1059. PMLR, June 2016. URL
mlr. wwl..prees//ee. wwwpp///s//seeeeeeeeeeeeeeeeeeee v48/ gall6. html
ISSN: 1938-7228.
Ariel Gera, Alon Halfon, Eyal Shnarch, Yotam Perlitz, Tiat Ein-Dor, and Noam Slonim. Zero-Shot Text Classification with Self- Trainin. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 1107-1119, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022 emnlp-main.73. URL atttss:/accaanthologg org/ 2022. emnlp waalllpmmai..... 73/
Mozhdeh Gheini, Xiang Ren, and Jonathan May. Attention is All You Need: Adapting Pretrained Transformers for Machine Translation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1754-1765, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v menpp..nnmm-m.............. 132. URL https:
//aclanthology.orgee
emnlp
1aa1pp-mmm wwmppinn...mmmm 132/
Tilmann Gneiting and Adrian E Raftery. Strictly Proper Scoring Rules, Prediction, and Estimation.
Journal of the American Statistical Association, 102(477):350-378 March 2007. ISSN 0162-1459, 1537- 274X. doi:
URL httpp://ww.. http://www.ool..//o/.o// ine com//ooi/bb//1..11111
0162145060000014477
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and et al. The Tlama 3 Herd of Models, November 2024. URL http:///rxii.
wwwpeiivvvggg///////e/e/// org/abs/2407. 240bb//1887 arXiv:2407.21788 [cs].
Shai Gretz, Alon Halfon, Ilya Shnayderman, Orith Goledo-Ronen, Artem Spector, Lena Dankin, Yannis Katsis, Ofir Arviv, Yoav Katz, Noam Slonim, and Liat Ein-Dor. Zero-shot Topical Text Classification with LLMs an Experimental Study. In Houda Bouamoor Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Lúnguistács: EMNLP 2023, pp. 9647-9676, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1 ht//:/1222222//iiii emnlp.647. fnd...g..-......6777.... URL https: //aclanthology.oo www.aatthhooooooooo22230.3 findings- (ammlp.6477)
Chuan Guo, Gcoff Pleiss, Yu Sun, and Kilian Q. Weinberger. On Calibration of Modern Neural Networks. In Proceedings of the 34th International Conference on Machine Learning, pp. 1321 1330. PMLR, July 2017. URL https:://prcceeeainnneee mlr.
http:///iww.......peeee/e/////// press//TT/gguo/7... html
Trevor Hastie, Robert Tibshirani, and Jerome Friedman. The Elements of Statistical Learning. Springer Series in Statistics. Springer New York Inc., New York, NY, USA, 2001.
19

---

Published in Transactions on Machine Learning Research (01/2026)
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly, Sarameter-Ffficient Transfer Learning for NLP. In Proceed- ings of the 36th International Conference on Machine Learning, pp. 2790 2799. PMLR, May 2019. URL attps://proceedingg mlr. pptps///9/hoolssyylasss. html ISSN: 2640-3498.
Yingxiang Huang, Wentao Li, Fima Macheret, Rodney A. Gabriel, and Lucila Ohno-Machado. A tutorial on calibration measurements and calibration models for clinical prediction models. Journal of the American Medical Informatics Association: JAMIA, 27(4):621 633, April 2020. ISSN 1527-974X. doi: 10.1093/
famia/ocz222
Mingjann Jiang, Yangjun Ruan, Sicong Huang, Saifei Liao, Silviu Pitis, Roger Baker Grosse, and Jimmy Ba. Calibrating Language Models via Augmented Prompt Ensembles. JCML 2023 Workshop Deployable Generative AI, June 2023. URL https://opperevieeee
Tom Joy, Francesco Pinto, Ser-Nam Lim, Philip H. S. Torr, and Puneet K. Dokania. Sample-lependent adaptive temperature scaling for improved calibration. In Proceedings of the Thirty-Severce AAAI Con- ference Artificial Intelligence and Thirly- Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, volume 37 of AAAT23/IAAT /EAAA'23, pp. 14919-14926. AAAT Press, 2023. ISBN 978-1-57733-880000 doi: 297112.26744
10.1609 10.1609/aaai..3371222 /tt///aaaa.v.1..227..... URL https: org/ 10. 1609/aa0a
Sanyam Kapoor, Nate Gruver, Manley Roberts, Arka Pal, Samuel Dooley, Micah Goldblum, and Andrew Wilson. Calibration-Tuning: Teaching Large Language Models to Know What They Don't Know. Tn Raúl Vázquez, Hande Colikkanat, Dennis Ulmer, Jörg Tiedemann, Swabha Swayamdipta, Wilker Aziz, Barbara Plank, Joris Baan, and Marie-Catherinn de Marneffe (eds.), Proceedings of the 1st Workshop on Uncertainty-Awwon NLP (UncertaiNLP 2024), pp. 1-14, St Julians, Malta, March 2024. Association for Computational Linguistics. URL https://aclanthonn
2024. uncertainlp-1.1/
Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. RemnonstrateeSSarch-Peeeeel Composing retrieval and language models for know snwwledge-intensive NLP, January 2023. URL
14024 arXiv:2212.1400 [cs].
Lorenz Kuhn, Yarin Gal, and Scbastian Farquhar. Semantic Uncertainty: Linguistic Invariances for Uncer- lainty Estimation in Natural Language Generation. In The Eleventh International Conference on Learning Representations, September 2022, URL https://openreviee: wttpe/vvieeinneffffrreeeeee http://rrummiid-/---APPPPt
Meelis Kull and Peter Flach. Novel Decompositions of Proper Scoring Rules for Classification: Score Ad- justment as Precursor to Calibration. In Machine Learning and Knowledge Discovery in Databases, pp. 68 85, Cham, 2015. Springer International Publishing. ISBN 978-3-319-23528-8. doi: 10.1007/ 978-3-319-22 22119-235888111 5.
Meelis Kull, Miquel Perello Nicto, Markus Kängsopp, Telmo Silva Filho, Hao Song, and Pe- (er Flach. Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with Dirichlet calibration. In Advonces in Newral Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL https://papers neurips. cetrrips.cc/aee cc/paper__... hilees/paper/211//haass Abstract. html
Ken Lang. NewsWeeder: learning to filter netnews. In Proceedings of the Twelfth International Conference on International Conference on Machine Learning, (CML'95, pp. 331-339 San Francisco, CA, USA, July 1995. Morgan Kaufmann Publishers Inc. ISBN 978-1-55860-37766
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N. Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick van Kleef, Sören Auer, and Christian Bizer. DBpedia large-scale, multilingual knowlodge base extracted from Wikipedia. Semantic Web, 6(2):167-195, Jamuary 2015. ISSN 1570-0844. doi: D0.3233/SW-140133 URL https://contenn: wwwpttennn.ioppessss.oo swapreess.com/attttiee///eaaoaa sw134 Publisher: IOS Press.
20

---

Published in Transactions on Machine Learning Research (01/2026)
Brian Lester, Rami A1-Rfou, and Noah Constant. The Power of Scale for Crrameter-Effiieen Prompt Tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045 3059, Online and Punta Cana, Dominican Republic, November 2021. Association for Compu- tational Linguistics. doi: 10.18653 00006553/1222// w1t20//.cmnln-m111.. emnlp-main.22 URL emnlp-main. 243/
hww.hathhooooooooooooooooo org/2021.
Chengzu Li, Han Zhou, Goran Glavaš, Anna Korhonen, and Ivan Vulié. Large Language Models are Miscali- brated In-Context Learners. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACT. 2025, pp- 11575-11596, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-8-89176-256-4 25--5.... doi: 10.18653/vl
acl.ing URL attps::/aclantthol.gg org/ f wwndinggnngiaa aaw.-ccl...00.. 603/
Yingjie Li, Yun Luo, Xiaotian Xie, and Yue Zhang. Task Calibration: Calibrating Large Language Models on Inference Tasks. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 6937-6951, Vienna, Austria, July 2025b. Association for Computational Linguistics. 20025..inding--cee
Linguutttionnallnnunsssssssoo ISBN 979-8-89176--56-5. doi: 10.18653/ URL httpss///acaaattooooooor/o////// orgg/2228..nnnnngs-acc.....
Vladislav Lialin, Vijeta Deshpande, Xiaowei Yao, and Anna Rumshisky. Scaling Down to Scale Up: A Guide to Drrameter-Effiiien Fine-Tunnin,, November 2024. URL
15647
arXiv:2303.15647 [cs].
Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10351- 10368, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/
e1/2002..enlllmmaii ma8n. URL attpss:/aclannthologg oww.ggg...r..ooo org/2024. 2024. emnlp-main.57 main.578/
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.
ACM Compul. Surv., 55(9):195:1 195:35, January 2023. ISSN 0360-0300. doi: 10. 10.1145/3560815 URL
https://dl. acm. /.0.../0////////.1111//555111111
Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, Low-Rank Adaptation. In Proceedings of the 41st International Conference on Machine Learning, pp. 32100-32121, PMLR, July 2024. URL https:// proceedings .mr.angg...rr...mm.............. mlr. ppress/v235/1u22250 html, ISSN: 2640-3498.
and Min-Hung Chen. DoRA:
Clara Meister, Elizabeth Salesky, and Ryan Cotterell. Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguisties, pp. 6870 6886, Online, July 2020. Association for Computational Linguistics. doi: 10.
awl.mmn..cmmmi.... 6615.
URL https::/accaattoolle wwwp://..r//202..../ mall-main... 615/
Marius Mosbach, Tiago Pimentel, Shauli Ravlogel, Dietrich Klakow, and Yanai Elazar. Few-shot Fine- tuning vs. In-context Learning: A Fair Comparison and Evaluation. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 12284-12314, Toronto, Canada, July 2023. Association for org/
Computational Linguisties. doi: 10.18653/v1 2.0.2...nning-----i.. 2023.f www.innnnnnncc.777 779/
URL
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarivation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brussels, Belgium, October 2018. Association for Computational Linguistics. doi: 10.18653 10.18653/v1/D11 1206. URL https: //aclanthology..o Saw.aatthoooogooor//------1
Reham Omar, Omij Mangukiya, Panos Kalnis, and Essam Mansour. ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbots, February 2023. URL
06466 arXiv:2302.06466 (cs).
21

---

Published in Transactions on Machine Learning Research (01/2026)
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko A Altenschmidt, Sam Altman, and et al. GPT-4 Technical Report, March 2024. URL arXiv:2303.0877 [cs].
http://arxiv.orrs httgg/ab/2300..000
John C. Platt. Probabilistic outputs for support vector machines and comparison to regularized likelihood methods. Advances in Large Margin Classifiers, 10, June 2000.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yançi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Unified Text-to- Text Transformer. Journal of Machine Learning Research, 21(140):1 67, 2020. ISSN 1533-7928. URL http://jmi owg. pttp:/ppprs//1/200-1220 074. html
Tim Schopf, Daniel Braun, and Florian Matthes. Evaluating Unsupervised Text Classification: Zero-shot and Sinilarity-based Approaches. In Proceedings of the 2022 6th International Conference on Natural Language Processing and Information Retrieval, NLPIR '22, pp. 6-15, New York, NY, USA, June 2023, Association for Computing Machinery. ISBN 978-1-4508-9962-9 doi: 10.1145/358276..555 35872795. URL
http:://ob..orr/1011145///////// https://doi.ors: orooooo//// 10. 11145//352276.3352222
Maohao Shen, Subhro Das, Kristjan Greenewald, Prasanna Sattigeri, Gregory W. Wornell, and Soumya
Ghosh. Thermometer: Towards Universal Calibration for Large Language Models. In Proceedings of the 41st International Conference on Machine Learning, pp. 44687 44711. PMLR, July 2024. URL https: //proceedings. mlr. wttpa////v////hee22..c html
ISSN: 2640-3498.
Chenglei Si, Chen Zhao, Sewon Min, and Jordan Boyd-Graber. Re-Examining Calibration: The Case of Question Answering- In Findings of the Association for Computational Linguisties: EMNLP 2022, pp- 2814 2829, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: (..18633/v1/202 w.n..ii...inns--.... 1ndings-emmip.2200 URL https://acclanbbooogg..ooo////// ftw.:og..fiiniin--eeeee 204/
Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Rantos-Rodriguez Meelis Kull, and Peter Flach. Classifier calibration: survey on how to assess and improve predicted class probabilities. Machine Learning, 112(9):3211 3260, Sep 2023. ISSN 1573-0565. doi: 10.1007 (00000//110099-02--63333--- URL https: //1ink.springen wwt./o/ooo//ccccocc www////ooneen/////////// 1t00://00000000/1119900000000000
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over Sentiment Tree- bank. In Proceedings of the 2013 Conference Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013, Association for Computational Linguistics. URL http://ccaattoooooyyrr////111110 org/D13-1170/
Dilara Soylu, Christopher Potts, and Omar Khattab. Fine- Mane-Tuning and Prompt Optimization: Two Great Steps that Work Better Together. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 10696-10710 Miami, Florida, USA, November 2024. Association for Compu- tational Linguistics. doi:
htt////1112222.m/////// 197... URL https://aclaathologg.. orgg:///ooooooooo.2222220.
emnlp 1ellp-mmin.500
Claudio Spicss, David Gros, Kunal Suresh Pai, Michael Pradel, Md Rafiqul Islam Rabin, Amin Alipour, Susmit Jha, Prem Devanbu, and Toufique Ahmed. Calibration and Correctness of Language Models for Code. In 2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE), pp. 495 507, IEEE Computer Society, October 2024. ISBN 979-8-3315-0560-1. doi: 10.1109//CSE555347222225 2020550033444220000000000 URL httpp://wwws computer org/
ISSN: 1525-
Elias Stengel-Eskin, Peter Hase, and Mohit Bansal. LACIE: Listener-Aware Finetuning for Calibration in Large Language Models. In The Thirty-eighth Annual Conference on Neural Information Processing
Systems, November 2024. URL http://////////oooooaaa////////a Dooerr/iZerrttaattooaaaaiiiiiiie B2nsal2)
22

---

Published in Transactions on Machine Learning Research (01/2026)
Katherine Tian, Eric Mitchell, Allan Zhou, Archit Sharma, Rafacl Rafailov, Huaxiu Yao, Chelsea Finn, and Christopher Manning. Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tumed with Human Feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. -443, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 20.18653/v1/20223 een.p-main.33330 org/2023. e.r1.mmmlllm... main..in...mm mm310.30/.
main.330. URL.
K. Van Hoorde, S. Van Huffel, D. Timmerman, T. Bourne, and B. Van Calster. A spline-based tool to assess and visualize the calibration of multiclass risk predictions. Journal of Biomedicall Informatics, 54:283 293, April 2015. ISSN 1532-0464. doi: 10. 11.1:/...b...1..122...
URL fttps://aw..sceencddirre com/
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, In Proceedings of the 2018 EMNLP Workshop Blackboz:LL: Analyzing and Interpreting Neural Networks for NLP, pp. 353- 355. Brussels, Belgium, November 2018. Association for Computational Linguistics. doi: 10.18653/v1/ W18-5446. URL https://aclaanthology/ org//18--4/66
Jason Wei and Kai Zou. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In Kentaro Imui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.). Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pp. 6382-63888 (882-6888, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. (0.18633/11/D19-16700 URL https:
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Lc. Finetuned Language Models are Zhro-Shot Learners. In International Conference on Learning Representations, October 2021. URL http:://pppaaeeeeeeeeeeerrrrrraa
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of Thought Prompting Elicits Reasoning in Large Language Models. In Advances in Neural Information Processing Systems, October 2022. URL https://openreviie http:/rrrwwweeeeeffouuu...m. VJQIMeSB_J
Robert L. Winkler and Allan H. Murphy. " "Good" Probability Assessors. Journal of Ayplied Meteorology and Climatology, October 1968. ISSN 1520-0450. URL https::/jjournall wwt.a//.....g/////////oaaaaaaaaa appme/7/5/152004450 11we000_11____000 111116880__0000 10011001116_000777______22222 ____0__o__2_____ xml
Johnathan Xie, Annie S Chen, Yoonho Lee, Eric Mitchell, and Chelsea Finn. Calibrating Language Models with Adaptive Temperature Scaling. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (cds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18128 18138, Miami, Florida, USA, November 2024, Association for Computational Linguistics. doi: 10.18653/
ht/2222..cmll-maaii mann-m00..00 URL https::/accaatthloog/ www.ttthhooooooooo///.o o02/ emnlp- main. 1007/
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian Hc, and Bryan Hooi. Can LLMs Exprcss Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs. In The Twelfth In- ternational Conference on Learning Representations, 2024. URL https://openreviiet wttpeeiiiiiieneeeeeeeeeeeeeeeeee mnetfforumiid g]eCKFxFpZ]
An Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, and et al. Qwen2.5 Technical Report, December 2024. URL http://arxiv... org/ 111/22221111111 1:X1v [es).
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R. Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Thirty-seventh Conference on Neurol Information Processing Systems, November 2023. URL https:
//
23

---

Published in Transactions on Machine Learning Research (01/2026)
Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking Zero-shot Text Classification: Datasets, Eval- uation and Entailment. Approach. In Kentaro Inui, Jing Jiang, Vincent. Ng, and Xiaojun Wan (eds.); Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) pp. 3914-3923, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/v1D1194 URL http://cclmmaoooog..og////////ss
Wonjin Yoon, Jinbyuk Lee, Donghyeon Kim, Minbyul Jeong, and Jaewoo Kang. Pre-trained Language Model for Biomedical Question Answering. In Machine Learning and Knowledge Discovery in Databases, pp. 727 740, Cham, 2020. Springer International Publishing. ISBN 978-9-030-43887-6. doi: 10.1007/ 008-3-030-3888-- 64.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791 4800, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10653/11/P19-14/ URL htttps:/aaclanthologg) org/P19-1472/
Taolin Zhang, Zerui Cai, Chengyu Wang, Minghui Qiu, Bite Yang, and Xiaofeng He. SMedBERT: A Knowledge-Fnhancee Pre-trained Language Model with Structured Semantics for Medical Text Mining. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5882 5893, Online, August 2021. Association for Computational Linguistics. doi: 10. 111.1//1////2//2 acc.o.cl-ooonn......... aal-lonng.477. URL https::/aalaanhhology .rceey..rrggree... wrtp//21.......oo org/2021.acl-lontt .40/o/g..5//// 457/
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification. In Advances in Newral Information Processing Systems, volume 28. Curran As- sociates, Inc., 2015. URL 255c105517773353a8cbde8777222222 Abstaaatttttttttttttataaaaaa html
http://ppooodding..urur.....co caaape_ffllspppeeeeearpspss www..///pppp//21////aaaa
Xuan Zhang, Navid Rajabi, Kevin Duh, and Philipp Koehn. Machine Translation with Large Language Models: Prompting, Few-shot Learning, and Fine-tuning with QLoRA. In Proceedings of the Fighth Con- ference on Machine Translation, pp. 468-481, Singapore, Docember 2023. Association for Computational Linguistics. doi: 10.18653/v1/ 10/5533/1202..wm/// 100022...mm1.44 1.43. URL htttp://acllathhologg woooor///000 2023. wmt- 1.43/
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate Before Use: Improving Few- shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning, pp. 12697-12706. PMLR, July 2021. URI. htttps://proceediiee mlr press/v139/ v130///a099//1e/ html ISSN: 2640-3498.
24

---

Published in Transactions on Machine Learning Research (01/2026)
A Dataset details
A.1
Used prompts
Table shows the structure of the prompt used as input to each model. Table 2 shows the instruction and list class identifiers for each dataset.
Model
pinnutt text,
LL3M33.2 <lbegin_otd of
Sebggin_o_ttxxlleee / /ttet//>>//stttt_heedee_tie www.hheeerridddyyyeeeeeeeee htggghhh___ttttaaaaaaseaaeaaeeer ear_nkled__daaeedeeeeeeee Serd__hadde____ idd>
instructions and task description<leot, (i___ddd>/sstarttt starsstttt_eeeer____1 / header_idd _id
imput text<|start, starrkeaaer__trrrrr___ id >assistant< /eattttttn///nndddd/ end header_id]
class identifier(y)
Qwen2.5
stin_start/eeyste
instructions and task pascription<lim_ent
"rim_star|>uuse
input
<lim_stard >>assistant class identifier(y) (y)
Table 1 Structure of the prompt for each model
A.2 Priors distribution
Figure 7 shows the prior distribution for each dataset. Classes were sorted by probability value. SST-2 and AGNews almost perfectly balanced. For DBPedia and 20Newsgroup, the ratio between the prior for the most frequent and for less frequent class is around 2, while for Banking77 is around 3.
D8pedia
Newsgroups
Banking77
04
03
032
300
04
Figure 7: Prior distribution for cach dataset. The class names not included in the x-axis since the main goal is to show the class imbalance in each set.
B Implementation details
When using SFT in all its variants, we applied LoRA reparametrization to all attention, feed forward layers, and embeddings matrices of the model. LoRA hyperparameters wore set to the default ones from the LitGPT library (Piropout = 0.05, = 8, a = 16). AdamW optimization was used, with a fixed learning rate of 10-3 and batch size of 8 for all datasets and adaptation sizes. Training and inference were done in bfloat16 precision.
For SFT-ES, we select the model for which the validation cross-entropy is minimum using early stopping with patience of 10. For SFT-FS-retrain we proceed as follows: we keep track of the numbers of training steps used for fine-tuning for SFTTES. Then, we performed second fine-tuning process with the same parameters and same number of steps, but using all available samples for training. Finally, for SFT we trained the
httpp////ggthubb.cmmmiggtnnnggg AI/1itgpt
25

---

Published in Transactions on Machine Learning Research (01/2026)
Dataset
instruction template
4 class-1dentifier(o))
SST-2
Determine if the following review is positive or neg- ative, based on the input given by the user.
Positive; Negotive
AG News
Determine the category of the news article given by the user.
World; Sports; Business; Science and Technology
DBPedia
Determine the category of the article given by the
Company; Educational Institution; Artist; Athlete; Office Holder;
Meon Of Tronsportation; Building; Natural Place; Village; Animal; Plant; Album; Film; Written Work
user.
20 20Newsgroups
Determine the category of the posted document given by the user.
Atheism; Graphics; Microsoft; IBM Hardware; Mac Hardware; X
Window System; Sales; Cors; Motoreyoles; Baseball; Hookeg; Cryp- tography; Electronics; Medicine; Space; Christianity; Guns; Middle East; Politics; Religion
Banking77
Classify the intent of the question input by the user.
Active my card; Age lmit; Apple pay or google pay: ATM support; Automatic top up; Balance not updated after bank transfer; Balance not updated after cheque or eash deposit; Benefictary not allowed; Cancel transfer; Card about to expire; Cord acceptance; Card or- rival; Cord delivery estimate; Card linking; Card not working; Card payment fee charged; Card pagment not recognised; Card payment wrong exchange rate; Card swallowed; Cash withdrawal charge; Cash withdrawal not recognised; Change pin; Compromised card; Contoct- less not working; Country support; Declined card poyment; Declined cash witthdrowwl; Declined transfer; Direct debit pagment. not recog- nised; Disposable cord limits; Edit personal details; Eachange charge; Ezchange rate; Exchange via app; Extra charge on statement; Failed transfer; Fiat currency support; Get disposable virtual cord; Get phys- teal card; Getting spare card; Getting virtual card; Lost stolen card; Lost stolen phone; Order physical card; Passoode forgotten; Pend- ing card payment; Pending cash withdrowal; Pending top up; Pend- ing transfer; Pin blocked; Reeciving money; Refond not showing up: Request refund; Reverted card payment?; Supported cards and cur- rencies; Terminate account; Top up by bank transfer charge; Top up by card charge; Top up by cash or cheque; Top up failed; Top up tim- its; Top up reverted; Topping up by card; Transaction charged twice; Transfer fee charged; Transfer into occount; Tranefer not received recipient; Transfer timing; Unable verify identity; Verify my iden- tity; Verify souree of fundss; Verify top Virtual card not working: Visa or mastercard; Why verify identity; Wrong amount of cash re- ceioed;; Wrong enchange rate for cash withdrrodd
Table 2: Instructions and list of class identifiers for each dataset.
model using all the available samples until the training cross-entropy converged using the same criterion as for SFT-ES.
We train all the PHC methods described Section 5.2 with the LBFGS algorithm a single batch per epoch until the training loss does not decrease its value for 10 consecutive steps. Learning rate was set 10-3 and the maximum number of iterations in each step of the LBFGS algorithm to 40.
26

---

Published in Transactions on Machine Learning Research (01/2026)
B.1 Training curves example
Figure shows one seed of the training curves for the SFT-ES method. It can be seen that the validation curve reaches convergence for the chosen patience value (10) for both sizes and all datasets. Note that the model with the best validation loss is selected after training is stopped. Other seeds show the same trend.
Triiiing curves SFT-ES for one seed LLaMAB 2-t.AA2---B--aa--- B-instract
DOPedis
Ranldnn77
Uailtation
Vadida
Valddiioon
BYY 25. Moatton
Thiin Loos
Valdetion
Figure 8: Training curves example for SFT-ES using a patience value of 10.
C Additional results
C.1 Additional metrics
One of the main concerns about the NCE is that the logarithmic loss is an unbounded metric which may allow a few extreme values to disproportionately affect the average. While, as we argue in Section 4 we believe this is desirable property of this metric, it interesting to analyze the impact of this behavior on the results. To this end, evaluated a subset of the methods using the Normalized Brier Score (NBS), which is another PSR that is also commonly used metric to assess the quality of posterior probabilities. The NBS, unlike the NCE, is bounded metric defined as:
NBS ΣN ΣK k= ( 1 (m))))))))
Σ N4(1 N₂/N)
Figure g shows the results for a subset adaptation methods for the LLaMA3.2 model for NBS and for NCE, for comparison. The results for NBS are qualitatively similar those for NCE. particular, the main conclusion from the paper that SFT PHC-TS is the method that leads the best results also holds for the NBS metric. One notable difference in conclusions between the two metries is that, while SFT is consistently worse than SFT-FS-retrain in terms of NCE, the two methods are comparable in terms of NBS. This is due to the fact that the overfitting that occurs with the SFT method results in overconfident systems that may produce extremely incorrect posterior values, which are much more heavily penalized by the NCE metric. This, in fact, supports our decision to use the NCE metric, since we wish to severely penalize systems that may produce extremely incorrect posteriors.
Figure also shows the Relative Calibration Loss (RCL) (Ferrer & Ramos 2025 and the ECE (Guo al., 2017), two metrics designed to measure model's degree of miscalibration. The RCL is given by the relative difference between the cross-entropy of the model and the cross-entropy that is obtained after applying a calibration transformation. For the results in this figure, the calibrator is given by the DP transform trained on the test set itself. The ECE is given by the absolute difference between the model's posteriors and the posteriors after calibration with histogram binning, also trained on the test data.
The calibration metrics show that the NoA system is greatly miscalibrated, with RCL values between 0.3 and 0.6 (between 30% and 60% of the cross-entropy value is due to miscalibration). The SFT system is also
27

---

Published in Transactions on Machine Learning Research (01/2026)
SST-2
AGNews
DBPedia
20 Newsgroups
Banking
NeA
Nok - 2.01
NoA 1,09
NoA 2.24
1.0
0.
0.3
0.25
0.7
0.8
0
0.2-
0.8-
0.6
0.35
0.15-
0.5
0.3
0.3-
0.4
0.25
0.05
0.4
0.2
0.3
32
64
126
25,6
512
0.2
128
258
512
0.0
56
1:2
234
448
846
0.3
80
160
320
840
12600
0.0
154
308
616
1232
2464
0.6
0.45
0.0
0.5
0.6-
0.7-
0.45
0.55
0.3
0.35-
0.3-
0.4
0.77
NoA 0.68
0.7
0.8-
NoA
1.37
0.5
0.3
0.2
0.2
0.3
0.85
0.1-
0.45
0.55
0.3-
0.4
52
64
12t
256
512
12
64
128
258
512
56
112
224
448
896
80
160
120
640
12400
0.3
154
308
616
1232
2464
0.6
0.5
0.2
0.45
0.5
0.5-
0.175
0.15
0.35
0.4
0.5
0.60
0.2
0.2
0.3
0.125
0.075
0.1-
0.25
0.15
0.2
0.34
0.2
0.1
0.08
0.1
0.3
64
128
258
512
12
128
256
112
224
896
80
160
320
640
1280
154
308
616
1232
2464
0.08
0.1
0.00
0.35
0.08
0.09
NiA 0.28
0.07
Nad Hovamovvak-
0.3
0.3
Nox
0.50
0.00
0.06
1 0.05
4 0.06
0.05
0.07
0.05
0.03+
0.05-
0.04-
0.20
0.2
0.2
0.04
0.02
0.00
0.03
0.06
0.05
0.00
512
112
Number samples
Method
NoA
SFT-ES-retrained
SFT
PHC-DP
SFT PHC-TS
Figure 9: Comparison of NCE, NBS, Relative Calibration Error and ECE metrics for a subset of adaptation methods.
greatly miscalibrated, except for the DBPedia datasct. The other models: SFT-ES-retrained, PHC-DP and the combined model SFT+PHC-TS are all relatively well-calibrated. In particular, the combined system has RCL values below 0.2 for all cases.
The ECE values generally agree with the RCL values, with some notable exceptions. For example, for DBPedia, the NoA system has an RCL of 0.52, indicating gross miscalibration (52% of the cross-entropy is due to miscalibration). Yet, the ECE is 0.13 which may appear relatively low. The discrepancy may be due both to the lack of normalization of the ECE metric and to a suboptimality in the histogram binning transform. An exhaustive discussion on the suboptimality of the ECE can be found in [eereer & Ramos 2025).
In the machine learning literature, it is very common to use calibration metrics instead of SPSRs assess the quality of posteriors. Figure is an example of why this is problematic practice. A system may be worse calibrated than another but have a lower cross-entropy. For example, while PHC-DP is close to perfectly calibrated for all datasets, its NCE is worse than the one of the SFFT-ES-retrained model. That is, SFT-ES-retrained has better-quality posteriors than PHC-DP despite them being worse calibrated. Hence, we should always use an SPSR to assess the quality a model's posteriors. Calibration metrics may be used analyze whether the posteriors can be improved by a post-hoc calibration stage, but the performance after the calibration stage is added should be judged based on an SPSR. An extensive discussion on these issues can be found in [Ferrer& & Ramos 2025).
C.2 Supplementary table for LLaMA3.2 model
Table 3 shows numerical values of the NCE and NER values for the methods shown in Figure values correspond to the left-most and right-most points of that figure.
These
28

---

Published in Transactions on Machine Learning Research (01/2026)
SST-2
AGNews
DBPedia
20 Newsgroups
Banking77
NCE
NER
NCE
NER
NCE
NER
NCE
NER
NCE
NER
NoA
0.80
0.54
1.49
0.60
2.01
0.58
1.09
0.61
2.24
0.88
SFT
1.00
0.18
0.46
0.17
0.06
0.04
0.88
0.50
1.05
0.46
SFT-ESSrotrrined
0.70
0.21
0.42
0.22
0.06
0.04
0.63
0.49
0.73
0.53
PHC-DP
0.80
0.56
0.57
0.41
0.38
0.32
0.57
0.54
0.60
0.57
SFT PHC-TS
0.60
0.18
0.30
0.17
0.06
0.04
0.51
0.50
0.50
0.46
SFT
0.68
0.15
0.35
0.13
0.02
0.01
0.57
0.32
0.13
0.13
8
SFT-ES--eeraiiee
0.43
0.17
0.25
0.16
0.02
0.01
0.37
0.35
0.15
0.17
PHC-DP
0.78
0.55
0.55
0.40
0.34
0.27
0.53
0.50
0.48
0.51
SFT PHC-TS
0.33
0.15
0.20
0.13
0.02
0.01
0.34
0.32
0.11
0.13
Table 3: Median value of NCE and NER for the methods shown in Figure corresponding to the LLaMA3.2
model for the smallest (T" = 4) and largest (T" = 8) adaptation size. The best performance for each task within the corresponding adaptation size is shown in bold.
C.3 Results for Qwen2.5
Figure 10 shows selection of the adaptation methods for the Qwen2.5 model. In addition, Figure 11 shows the same results as Figure 5 but for Qwen2.5. As explained in Section 6.6 this model shows better performance than LLaMA3.2 before adaptation and similar performance after adaptation with the proposed method. Note that the NER curves for SST-2 appear noisy but are almost constant (note the range of the axis). Finally, Table is equivalent to Table but for the Qwen2.5 model.
DBPedia
Newsgroups
Banking77
1.4
2.22
1.82
3.82
NCE
0.8
0.6
0.4
0.8
0.6
NER
0.4
0.2
0.0
N
32
512
N 32
N 512
N = 56
896
N 80
1280
154
N 2464
Adaptation sizes
Method
SFT
PHC-TS
SFT PHC.DP
NoA
SFT-ES SFT-ES-retrained
PHC-DP
SFT PHC-TS
PHC.VS
SFT PHC.VS
Figure 10: NCE and NER for all adaptation methods and two different adaptation samples (T" and I' 8) on Qwen2.5 model. Bars are divided by groups of colors: green for SFT, red for PHC and blue for combined methods. The bar height corresponds to the median across seeds used to select adaptation samples, and confidence intervals (black vertical lines top of the bars) correspond to the 1st and 3rd quartiles.
29

---

Published in Transactions on Machine Learning Research (01/2026)
SST-2
AGNews
DBPedia
20 Newsgroups
Banking77
0.5
0.4
64
0.9
035
NoA 0.7
NoA 2.22
0.65
NoA. 1.82
0.8
NoA 3,62
0.7
0.25
NCE
0.75
0.75
5/2
0225
0.25
0.3
0.35
0.2
0.175
0.3
0.1
0.15
32
64
12.
256
512
0:35
32
64
128
256
512
6.0
56
112
224
468
896
0.25
80
160
320
640
1280
0.0
154
308
616
1232
2464
0.12
0.24
0.5
0.5
0.7
NoA 0.27
NoA 0.39
NoA 0.80
NoA 0.86
0.115
0.25
0.6-
02-
NER
0.12
0.38
2.45
0.35
0.4
5.1
0.35
0.3
6:12
0.3
0.2
32
128
256
512
64
128
256
512
56
112
224
468
896
0.23
80
160
320
640
12:0
0.1
154
308
616
1232
2464
Number of train samples
Method
NoA
SFT-ES-retrained
SFT
PHC-DP
SFT + PHC-TS
Figure 11: Performance of selected adaptation methods versus the number of adaptation samples on Qwen2.5. Note that the NER curve for SFT is behind the one for SFT PHC-TS, since TS does not change the NER, value. The NER curves for SST-2 appear noisy but almost constant (note the range of the y axis).
SST-2
AGNews
DBPedia
20 Newsgroups NCE
Banking77
NCE
NER
NCE
NER
NCE
NER
NER
NCE
NER
NoA
0.32
0.11
0.78
0.27
2.22
0.39
1.82
0.80
3.62
0.86
SFT
0.45
0.11
0.34
0.15
0.02
0.02
0.68
0.40
0.84
0.38
SPT-ES-retrained
0.44
0.12
0.31
0.19
0.03
0.03
0.49
0.39
0.76
0.56
PHC-DP
0.43
0.11
0.38
0.23
0.36
0.29
0.67
0.59
0.69
0.65
SFT PHC-TS
0.53
0.11
0.28
0.15
0.02
0.02
0.42
0.40
0.39
0.38
SFT
0.40
0.10
0.38
0.10
0.02
0.01
0.49
0.27
0.12
0.11
rer--ne-etrained
0.22
0.10
0.20
0.13
0.02
0.01
0.30
0.30
0.13
0.15
PHC-DP
0.20
0.11
0.36
0.21
0.30
0.25
0.62
0.54
0.58
0.56
SFT PHC-TS
0.20
0.10
0.19
0.10
0.01
0.01
0.29
0.27
0.10
0.11
Table 4: Median value of NCE and NER for the methods shown in Figure
11
corresponding to the Qwen2.5
model for the smallest (7" 4) and largest (T" = =8) adaptation size. The best performance for each task within the corresponding adaplation size is shown in bold.
30


Reviews:
[{"review_id": "dvE8WB8opx", "reviewer": "Reviewer_HgzT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary of contributions.**\n\nThe authors investigate combining the post-hoc calibration with the parameter-efficient supervised fine-tuning (PEFT) for generative language models (GLMs) to gain reliable posterior distributions. They claim that the PEFT leads to overfitting for GLMs on adaptation data and propose to apply a calibration method to outputs from fine-tuned models. In addition, they discover that applying temperature scaling calibration, a straightforward and widely known calibration method, would lead to poor quality of the fine-tuned model's posteriors, so they study different adaptation data splitting strategies for PEFT and calibration. Experiment results demonstrate that the well-designed adaptation data splitting strategy could consistently enhance a GLM's posterior distribution.\n\n**Strength.**\n1. The idea is straightforward and effective for GLMs on text classifications.\n2. This work reveals the impact of adaptation data on the post-hoc calibration method with detailed experiments.\n\n**Weakness & detailed comments.**\n\nAbout the investigation on the PEFT (LoRA SFT), which would lead to overfitting for GLMs on adaptation data.\n\n1. iiuc, the early stopping step is decided by Eq. (5) for **SFT-w-val** and **SFT-retrain**. Why not use the evaluation metrics Eq. (3) and Eq. (4) to choose the early stopping step?\n2. Follow 1. Besides the early stopping to prevent overfitting, I suggest the authors should study other regularization techniques, such as  L2 regularization [1], Entropy regularization (confidence penalties) [2], which directly address the overfitting issues. Some representative techniques suggested in the survey paper are also encouraged to add [3].\n3. Follow 1., I suggest revealing the early stopping results to check the choice of the patience number (10).\n\nAbout Post-hoc calibration (PHC) and temperature scaling (TS).\n\n4. I suggest unifying the form in PHC-X by the matrix or vector forms. For example, PHC-VS $\\mathbf{A}$ -> $\\mathbf{D}$ for the diagonal matrix; PHC-DP $\\alpha \\mathbf{I}$; PHC-DP $\\mathbf{I}$.\n5. \"The general expression for the affine transform, with A being a full matrix leads to unstable results and is very prone to overfitting given a relatively small amount of adaptation samples.\" -> First, why do we need to redesign the form $\\mathbf{A}$ from the scaler form in [4]. Second, what is *overfitting* of the softmax transform here?\n6. Follow 5., if we also encounter the overfitting issue in PHC, how do we prevent it in scenarios with a small amount of adaptation data?\n7. Based on this paper, focusing on Platt scaling for the post-hoc calibration, I suggest adding related works such as Dirichlet calibration [5] and Sample-Dependent Adaptive TS [6] to have a diverse and recent view of the calibration method.\n\nAbout Combining PHC and SFT\n\n8. The authors present the critical issue of selecting the proper adaptation for the SFT and PHC ('miscalibration that occurs on samples unseen during fine-tuning, i.e., samples with a score distribution that resembles the one we will see during deployment.'). However, how do we ensure that the held-out subset of the adaptation set for PHC is unbiased for the calibration in the deployment? Could you provide the theoretical bounds or the empirical experiences/discussions about this?\n9. Is that possible to utilize the K-Fold approach for obtaining a better calibrator? For example, use 5-fold holding-out subsets to obtain 5 calibrators for Step 4.\n\nAbout presentations.\n\n10. As mentioned in 8., the evidence of producing the reliable class probabilities might not be enough for me. I consider that 'Reliable' should be robust for arbitrary scenarios, such as out-of-distribution tasks or distribution shift (domain shift). Maybe the authors could consider the title like 'Empirical Study of Enhancing the Quality of the Posteriors for Generative Language Models in Classification Tasks'\n11. The motivation for selecting post-hoc calibration would be strengthened. For example, if we know the overfitting of the SFT fine-tuned models, why do we choose the post-hoc calibration instead of the regularization methods? Please try to highlight the motivation and rephrase in Section 1, Paragraph 4.\n12. While the authors mention that they do not use expected calibration error (ECE) because of the previous work ('ECE and other calibration metrics do not adequately address the problem of assessing the value of posteriors'), I consider that the reveal of the ECE is still valuable for assessing the calibration performance for the novel task such as LLM for classification.\n\n- [1] On calibration of modern neural networks (Guo et al., 2017)\n- [2] Regularizing neural networks by penalizing confident output distributions (Pereyra et al., 2017)\n- [3] Calibration in deep learning: A survey of the state-of-the-art (Wang, 2023)\n- [4] Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration (Ferrer & Ramos, 2025)\n- [5] Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration (Kull et al., 2019)\n- [6] Sample-dependent adaptive temperature scaling for improved calibration (T Joy et al. 2023)", "claims_and_evidence_yn": "No", "claims_explanation": "1. **Claim 1.** They propose to apply a calibration method to outputs from fine-tuned models to mitigate the overfitting brought by PEFT. However, they only test on the Post-Hoc Calibration instead of other calibration methods for the PEFT, which seems overclaim to me.\n2. **Claim 2.** \"Reliable\" for the PHC. As mentioned in Weakness & detailed comments, Point 6, this work misses the investigation to ensure the calibration would be stable.\n3. **Claim 3.** \"Reliable\" for the adaptation set. As mentioned in Weakness & detailed comments, Point 8, this work lacks the investigation into the properties of the held-out subset that impact the final results.", "audience_yn": "Yes", "audience_explanation": "1. The main contribution of this work is revealing the impact of adaptation data on the post-hoc calibration (PHC) method with detailed experiments and studying the combination of the PHC and supervised fine-tuning (SFT) for the generative language models. I think that it is valuable and interesting to the researcher who wants to apply the generative models to different domains.", "requested_changes": "1. Please at least add the motivation and discussions for selecting Post-Hoc Calibration instead of other calibration methods for the PEFT. Or if you can include and compare other calibration methods would be better.\n2. Please reveal more results to ensure the calibration is stable. For example, please clarify what 'overfitting' is for the calibrator. How do we identify/evaluate the issues of our clabrator?\n3. Please reveal the properties of the held-out subset of the adaptation set for SFT and PHC. For example, the similarity of distributions between two sets or SFT/PHC performances on each class.\n4. I still suggest revealing the ECE of your results. Please reveal ECE or give stronger reasons to convince me not to do that.\n5. Please identify the position of this work. You may consider the preliminary study of applying the Post-Hoc Calibration method for PEFT or \"reliable\" for the proposing methods of LLM for diverse scenarios.", "broader_impact_concerns": "As mentioned in previous parts about TMLR's audience, I think if this paper were extended to enhance the properties of LLM's output for arbitrary scenarios, it would be impactful to cross-domain applications.", "additional_comments": "Dear authors, reviewers, and the action editor, I apologize for my late reply to this work. I will ensure my engagement in the following discussions.\nSincerely, reviewer HgzT", "extra_scores": {}, "extra_text": {}}, {"review_id": "qXGIwxdR6O", "reviewer": "Reviewer_e5RS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper focuses on improving the reliability of class posterior probabilities generated by large generative language models (GLMs) for text classification tasks, particularly in low-resource scenarios. Key contributions include:  \n\n1. **Identifying Limitations of Existing Methods**:  \n   - The naive approach of computing class posteriors from GLM token probabilities yields poor results.  \n   - Parameter-efficient supervised fine-tuning (SFT, e.g., LoRA) improves decision quality but leads to suboptimal posteriors due to overfitting.  \n   - Common post-hoc calibration (PHC) methods like temperature scaling are ineffective for GLMs, while affine transformations (e.g., direction-preserving calibration) perform better.  \n\n2. **Proposing a Combined Approach**:  \n   A three-stage strategy combining SFT and PHC is developed to address overfitting in SFT and enhance posterior quality:  \n   1. Train SFT on a subset of adaptation data, holding out samples.  \n   2. Train a PHC calibrator on the held-out samples using posteriors from the SFT model.  \n   3. Re-train SFT on all data and apply the pre-trained PHC calibrator.  \n   This method outperforms standalone SFT or PHC in both posterior quality (measured by cross-entropy) and decision quality (measured by error rate).  \n\n3. **Evaluation Insights**:  \n   - Proper scoring rules (e.g., cross-entropy) are more reliable than calibration metrics like ECE for assessing posterior quality.  \n   - After adaptation, smaller GLMs (e.g., LLaMA3.2-1B) can match the performance of larger models (e.g., Qwen2.5-7B), suggesting the approach makes suboptimal models competitive.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides detailed experimental setups, including:  \n- **Diverse Datasets**: Five text classification datasets (SST2, AGNews, DBPedia, 20Newsgroups, Banking77) with varying class counts and imbalance levels, ensuring generalizability.  \n- **Clear Methodology**: Precise descriptions of adaptation methods (SFT variants, PHC techniques) and evaluation metrics (Normalized Cross-Entropy [NCE] for posterior quality, Normalized Error Rate [NER] for decision quality), with mathematical formulations and implementation details (e.g., LoRA hyperparameters, training procedures) in appendices.  \n- **Controlled Variables**: Experiments systematically vary adaptation data size (via \\(T'\\) parameterization) and compare methods under identical conditions (e.g., same models, seeds, and prompt templates), ensuring results are reproducible.", "audience_yn": "Yes", "audience_explanation": "The paper addresses pressing challenges in LLM reliability, low-resource adaptation, and evaluation—all of which are central to TMLR’s focus. It will engage researchers in areas like natural language processing, calibration of probabilistic models, and efficient fine-tuning, making it relevant to a meaningful segment of the journal’s audience.", "requested_changes": "Weaknesses:\n\n1. **NCE's sensitivity to extreme values**: Since NCE is unbounded, very high NCE values from a few extreme samples might affect the average result. It would be helpful to check if such cases significantly impact the overall evaluation.  \n\n2. **Clarity in Figure 1**: Figure 1 shows a sample prompt but doesn’t explicitly highlight the specific tokens used to calculate the posterior. Adding these tokens could make the posterior computation process easier to follow.  \n\n3. **Rationale for cross-entropy**: While cross-entropy is a key metric here, a bit more explanation (beyond citations) on why it’s well-suited for assessing posterior quality in this context would strengthen the core argument.  \n\n4. **Model comparison choices**: Comparing a small LLaMA model with a larger Qwen model (different architectures) makes it hard to isolate the effect of the adaptation method. A brief note on this choice would help readers interpret the results.  \n\n\nChanges:\n\n1. A quick check on how extreme NCE values affect the averages—this would confirm the metric’s reliability without altering the main findings.  \n\n2. Adding the specific tokens used for posterior calculation to Figure 1 would make the example more intuitive.  \n\n3. A short, straightforward explanation of why cross-entropy is ideal here (e.g., how it captures posterior quality for high-stakes tasks) would reinforce the metric’s relevance.  \n\n4. A brief note on why Qwen was chosen for comparison (e.g., to show generalizability across models) would help contextualize the model size/architecture differences.  \n\nThese tweaks would enhance clarity and strengthen the work, while preserving its core contributions.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LvMlPzq83j", "reviewer": "Reviewer_KMHy", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The main contribution of this paper is proposing a method combining SFT and PHC to improve the quality of posterior probabilities when using GLM to deal with text classification problems. During SFT and PHC process, different datasets are used. The feasibility of this method is validated experimentally.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Mostly clear, though some changes are encouraged.", "audience_yn": "Yes", "audience_explanation": "The proposed method enables a small LLaMA-1B model to achieve posterior probability performance in text classification comparable to that of a significantly larger Qwen-7B.", "requested_changes": "1. In the abstract, the first half should also emphasize that this paper focuses on text classification. In its current form, it may give the misleading impression that the work can produce posterior probabilities for answer accuracy in non-text-classification tasks. \n\n2. The main contribution of this paper does not actually include Input–Output prompting. Although some related techniques are used, mentioning it in the related works section seems unnecessary.\n\n3. The authors’ proposed method combines SFT and PHC, and the comparative experiments are designed as SFT-only, PHC-only, and the combination of SFT and PHC. However, both SFT and PHC are fairly standard and commonly used methods. For the posterior probability estimation problem in text classification that the authors aim to address, are there any existing alternative methods that could be included for comparison?", "broader_impact_concerns": "n/a", "additional_comments": "n/a", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)