Ensemble Debiasing Across Class and Sample Levels for Fairer Prompting Accuracy
Authors: Ruixi Lin
Ziqiao Wang
Yang You
Venue: COLM 2025 (2025)
Keywords: ensemble debiasing
accuracy imbalance
Heaviside step function
post-hoc correction
TL;DR: We propose a Heaviside step function based ensemble debiasing method, to flexibly rectify biased ICL output class probabilities across both class and sample levels, achieving fairer prompting accuracy for LLMs.

Abstract: Language models are strong few-shot learners and achieve good overall accuracy in text classification tasks, masking the fact that their results suffer from great class accuracy imbalance. We believe that the pursuit of overall accuracy should not come from enriching the strong classes, but from raising up the weak ones. To address the imbalance, we propose a Heaviside step function based ensemble debiasing method, which enables flexible rectifications of in-context learned class probabilities at both class and sample levels. Evaluations with Llama-2-13B on seven text classification benchmarks show that our approach achieves state-of-the-art overall accuracy gains with balanced class accuracies. More importantly, we perform analyses on the resulted probability correction scheme, showing that sample-level corrections are necessary to elevate weak classes. Due to effectively correcting weak classes, our method also brings significant performance gains to a larger model variant, Llama-2-70B, especially on a biomedical domain task, further demonstrating the necessity of ensemble debiasing at both levels. Our source code is available at https://github.com/NUS-HPC-AI-Lab/DCS.

Published as a conference paper at COLM 2025
Ensemble Debiasing Across Class and Sample Levels for Fairer Prompting Accuracy
Ziqiao Wang Department of Computer Science National University of Singapore ruixi@comp. nus, edd. sg 3734499.0u. nus, edu youyêcomp, nus. edu.
Ruixi Lin
Yang You
edu
Abstract
Language models are strong few-shot learners and achieve good overall accuracy in text classification tasks, masking the fact that their results suffer from great class accuracy imbalance. We believe that the pursuit of overall accuracy should not come from enriching the strong classes, but from rais- ing up the weak ones. To address the imbalance, we propose Heaviside step function based ensemble debiasing method, which enables flexible rec- tifications of in-context learned class probabilities at both class and sample levels. Evaluations with Llama-2-13B on seven text classification bench- marks show that our approach achieves state-of-the-art overall accuracy gains with balanced class accuracies. More importantly, we perform analy- ses on the resulted probability correction scheme, showing that sample-level corrections are necessary to elevate weak classes. Due effectively correct- ing weak classes, our method also brings significant performance gains to larger model variant, Llama-2-70B, especially on a biomedical domain task, further demonstrating the necessity of ensemble debiasing at both levels. Our source code is available at https: / //i thub. com/NUS AI-C-AIC-LLLL.S.
1 Introduction
Language models are good few-shot learners, but prompting can result in inevitable class accuracy imbalance. A language model can perform significantly better on certain classes compared to others, leading to biased performances across different classes (illustrated in Figure 1). This issue is particularly relevant in tasks like classification, or any scenario where the model must distinguish between multiple categories. For example, DBpedia ontology classification (Auer et al., 2007), the overall accuracy for 14 classes can reach 88% when prompting Llama-2-11 (Touvron et al., 2023), but class Nature's accuracy is 28%. Broadly, for open-ended generation tasks, when treating each token in the vocabulary as a distinct class, generating the most likely next token can be alternatively viewed as classifying over the entire vocabulary or soft labels (Thrampoulidis, 2024; Zhao et al., 2024), also susceptible to class accuracy imbalance. Therefore, accuracy imbalance prevalent and affecting users of language models, where classification is the most basic case manifesting such imbalances.
In text classifications, the prevalence of class accuracy imbalance sometimes hidden by single or dominant evaluation metric, such as overall accuracy. Especially for relatively large models, the fact that they excel on benchmarks which emphasize overall performances may hide poor performance in some classes. To enable a focused evaluation of whether language model performances are balanced across classes, the COBias metric was proposed (Lin & You, 2024a) to dedicate evaluating pairwise class accuracy differences.
The class accuracy imbalance issue is not particularly easy to solve from the root. Its causes come from within the model architecture. In details, language models rely on priors learned during training to make predictions. However, built on self-attention, the models are prone to the copying mechanism of the induction heads (Elhage et al., 2021; Olsson et al., 2022).

---

Published as a conference paper at COLM 2025
These heads predict repeated sequences, which can over-capture surface-level patterns in the data, leading to biased priors, and then biased predictions. Therefore, if certain classes concepts were more frequently emphasized in the training data, the model may disproportionately favor them (Zhao et al., 2021). Although a systemic change at model level or selecting high-quality data may help, it can be inefficient.
Besides what is rooted in the model and data, the way prompts are phrased can further bias the model towards certain classes (Lin & Ng, 2023; Ceballos-Arroyo
ACC.
Overall ... Debiesed
Class
Debiased
al., 2024; Sclar et al.,
Boc.
ICL
acc.
2024). In few-shot prompting, imbalanced class accuracy can be exacerbated because of under- represented classes in the demonstrations of prompt. Nevertheless, it only helps to a limited extent by making all classes well-representee in the demonstrations (Lin & You, 2024a). In addi- tion, Chain-of- Though style outputs may facili- tate self-correcting responses without relying on prompt engineering (Wang & Zhou, 2024), but the outcome is guided and restricted by reward signals. Lastly, although certain types of biases, such as gender bias or truthfulness, can be traced back to Transformer layers and intervened at inference time (Vig et al., 2020; Geiger et al., 2021; Li et al., 2023), interventions can be inefficient mitigate the accuracy bias, especially when it could be better treated without altering of the model's 's internal structure.
Class
2
3
8
7
8 9
10
Figure 1: The accuracy imbalance and what we can achieve with debiasing
Therefore, to effectively and efficiently reduce class accuracy imbalances, post-hoc optimiza- tion methods are introduced. As such, post-hoc probability correction complements training and prompting strategies to prevent the model from overly relying on learned patterns, and to incorporate necessary information in downstream tasks. This type of correction involves correcting In-context learned (ICL) output probabilities in inference time by combinatorially optimizing relatively small mathematical model. These direct rectifications are useful, because they can directly correct outputs, no matter what biases are encoded in the model prior, what prompts are used.
Related Work. Prior post-hoc correction work has investigated re-weighting ICL output probabilities through combinatorial optimization addressing COBias(Lin & You, 2024a;b) or calibration addressing biased predictions (Zhou et al., 2024; Zhao et al., 2021). The former uses labeled instances for optimization while the latter does not. Both lines of work advance overall classification accuracy with low costs and the flexibility of no language model parameter updates. The performance difference is that the combinatorial optimization works achieve better balanced class performances while improving overall accuracy. We follow the combinatorial optimization perspective for our ensemble debiasing.
For a background on combinatorial optimization, it provides a mathematical framework for maximizing or minimizing an objective function of decision variables, subject to constraints on some or all of the variables, involving inequalities, equalities, and integrality restrictions (Nemhauser & Wolsey, 1988). This framework essentially transforms a rich variety of tasks into discrete optimization models, which is robust and remarkably versatile. It solves real-world problems such as planning, scheduling, etc. in operations research, and has diverse applications in NLP, from structured prediction such as named entity recognition and dependency parsing, to reading comprehension, then to grammatical error correction, and to align LLM outputs with human preferences (Martins et al., 2009; Wang et al., 2010; Rush et al., 2010; Koo et al., 2010; Goldwasser et al., 2012; Wu & Ng, 2013; Berant et al., 2014; Roth Srikumar, 2017; Lin & Ng, 2021; Dakle et al., 2023; Zhang et al., 2024; Srikumar Roth, 2023; I. Garmendia et al., 2024). The core use of the framework is to translate a problem into a mathematical statement of one of the forms of linear mixed-integer programming problem (MIP), integer programming problem (IP), nonlinear integer programming problem (NIP), or other types (Roth & Yih, 2004; 2005; Hemmecke et al., 2010; Srikumar & Roth, 2023). Solving combinatorial optimization models is not an easy task. Among which, NIP is highly challenging, because the NIP model can contain discontinuous functions and is
2

---

Published as a conference paper at COLM 2025
inherently non-differentiable. Lin & You (2024a;b) proposed to address the search difficulty with metaheuristics, which we follow (more discussions in Appendix A).
Back to the combinatorial optimization based debiasing methods, their drawbacks lie in the single consideration at either class level (Lin & You, 2024a) or sample level (Lin & You, 2024b). In (Lin & You, 2024a), any sample gets re-weighted by the same set of per-class correction weights, making it hard to capture sample-level variances, which is particularly needed for weak classes. In (Lin & You, 2024b), corrections are made at sample level by assigning per-sample per-class correction weights that are computed by membership functions, however, capturing too much nuance may over-corrects strong classes.
To overcome the insufficiency in previous works, we propose meticulous post-hoc correc- tion framework DCS (Debiasing at Class and Sample levels). DCS an organic integration of broad and fine-grain fine-grained corrections across class and sample levels, pushing forward fairer prompting accuracy for LLMs. DCS automatically detects which classes need corrections that account for sample-level differences and which do not. In the following, we first intro- duce the Heaviside step function based ensembling strategy for DCS (Section 2) and the overall framework (Section 3), followed by quantitative evaluations and in-depth analyses (Section 4). In Section 5, we conclude with prospective applications and open problems.
Our contributions are summarized as below:
Post-hoc debiasing unifies class and sample level corrections. We ensemble weight correction and membership correction to enable rectifications of class probabilities at both class and sample levels, to enhance the performance of LLMs directly from their outputs.
SoTA scores on overall accuracy while maintaining low COBias DCS can achieve leading overall accuracy improvements while good COBias reduction across seven benchmark text classification tasks compared to the previous SoTA.
Optimal solutions need weight correction elevate weak classes. Quantitative analyses show that the average ratio of classes using membership correction to weight correction is between 0 and 1, where most classes apply weight corrections but lowest-accuracy (weakest) classes more often apply membership corrections, suggesting the necessity of keeping membership correction as an option and letting the optimization process determine if it is needed for any particular class.
2 Using Heaviside Step Function to Ensemble Class and Sample Level Debiasing
We introduce robust ensemble debiasing framework across class and sample levels, which is targeted specifically for balancing class accuracy while improving overall accuracy over individual methods. To achieve ensembling, a Heaviside step function seamlessly unifies two unique ICL output correction methods, bringing together the advantages of weight coefficient and fuzzy rule based corrections. This unified framework captures the intricacies of ICL output class probabilities with greater precision, reaching solutions that elevate weak classes while maintaining the strong ones. List of notations are presented in Appendix B.
2.1 The Mapping Function for ICL Probability Correction
As mentioned, common method for correcting probabilities involves multiplying each class probability by a coefficient (Lin & You, 2024a). This is a class-level correction. For an N-class task, we use a set of weights to adjust each dimension of the ICL output N- dimensional probabilities without considering the variations in the initial probabilities across different instances. In contrast, a more refined correction method employs membership functions from fuzzy logic (Lin & You, 2024b), where mapping function is applied to adjust each dimension. Different classes use different mapping functions. As result, class probabilities of different instances undergo varying degrees of transformation some are adjusted significantly, others minimally, and some remain unchanged, reflecting a finer, sample-level correction approach.
3

---

Published as a conference paper at COLM 2025
We propose enabling model-driven selection between coarse (class-level) and finer (sample- level) corrections in the optimization process, promoting efficient and precise post-hoc corrections for any class. Let's denote the probability of the m-th instance's i-th class as Pmi- We correct the per-sample per-class probability by mapping function fi P'm + ( Pmi E [o, 1]. To achieve our goal, the set of mapping functions Y combine weight correction with membership function correction, i.e, Y = IR Weigtt U Membershhip where IR Weight (wi,..., WWD and R Membership {M1,..., ). To dynamically choose the correction, we first introduce the Heaviside step function:
1, x x 0, otherwise
H(x)
(1)
Let E; denote the selection variable for class i, each č will be optimized to choose most suitable correction from Y. We conventionally note when č is optimized to be one of the indices from (1, DF}, the i-th class probability of each instance gets a personalized adjustment by membership function determined by Çü when ? is one of the indices from + I,.,, Dy + Dw}, , the i-th class probability of any input instance is corrected by weight coefficient determined by Şi Consequently, when Čj = k and k < Dp, the triangular membership function (following (Lin & You, 2024b)) based correction function μt is selected, the updated probability is computed by:
0,
Pmd S
Pmi ak
ag < Pmi S bk
bk 0k
(e(Pmm)
(2)
Ck Pmi
by < Pmi S Ck
CK bk
otherwise
Special cases: when Ag = by = O, H2 (Pmi)
= 9 Pmi for O S Pmi S C2 and O for Pmi 2 Cgi
when by Ck 1, (Pmmi) P A for ak dk
Pmi and for Pmi ak-
Otherwise, the weight correction function wp selected, and the updated probability is obtained by:
k k- DF
(x(Pmi)
Dw Pmi
(3)
Therefore, the per-sample per-class probability Pmi is precisely updated as follows.
P'm (P = mt (Pm:): H(Dp - (:) + w (pmi) H(ū: - DF -)
(4)
This results in more accurate correction for the ICL output probability, due to applying flexible fixes at either the broad level or the fine-grained level.
2.2 Mathematical Model
The mathematical model used to drive the selection based on the nonlinear integer pro- gramming model in (Lin & You, 2024a) with cecuracy-maximizing and COBias-minimiiin objectives. The combination of weight and membership corrections that optimizes the model's objective function will be selected. In short, the model is:
min = Err + BZCBsias (3) + TZPMI
s.t. Ž ( E (1,..., DF + Dw]
(5)
where E 7COBias 7.MI respectively corresponds to the overall error rate, COBias of all class pairs, and point-wise mutual information (PMT) between predicted and ground-truth classes (Appendix C). During optimization, at the beginning of each simulated annealing iteration, candidate selection č/'s s are generated and the correction functions fi' correct
each sample in the optimization set, yielding corrected predictions {fm 1M m= Each of the individual term in Z is essentially evaluation score computed using the corrected
4

---

Published as a conference paper at COLM 2025
Optimization Flowchart
ICL Class probabiities may need correction
Instance's probabiliies over N dasses
opt.
P11
Pra
P14
PrN
NIP
Updareed probe
P'1
10
P
Weight correction funcioon
Membership correction functions
Objective Aunction
Aoc
ZCBBis
ZPMI
Tune hyperpameterr
SA Algorithm (wth intial solution)
Generate a neighborhood solution
Metropolis Criterion Procegi c-///T
Setielactory dew sel?
Update temperature
No
Evaluate
Accept with probability
Yes
No
Outer-Ipop Stopping condition
Save solution
Better?
No
Inner-loop stopping condition?
Yes
Correct instances at inference time
Accept new solution
Infer on dev
Figure 2:
Overall framework of DCS
predictions and ground-truth labels { If the new Z is smaller than the current Z, the candidate selection is accepted and the best selection is updated; otherwise, the algorithm decides whether to accept the new selection based on the acceptance probability. Iterations repeat until stopping criterion is met. The optimization process will be elaborated in Section
Model parameters. This is a budget-friendly model with a total number of parameters (here we refer to search space size) of N(D; + Dw)-
Simulated Annealing for Solving DCS
Due to the non-differentiable nature the math model, we turn heuristics and employ the simulated annealing (SA) algorithm to solve the DCS model. The SA algorithm was proposed by (Kirkpatrick et al., 1983; Cerny, 1985). The forerunners successfully introduced the concept of annealing into combinatorial optimization, inspired by the annealing process of solid materials. starts at an initial temperature and cools down at a specified rate. As the temperature gradually decreases, the algorithm leverages probabilistic jumps to explore the solution space randomly, searching for the global optimum of the objective function, so can probabilistically escape local optima and eventually converge to the global optimum (Steinbrunn et al., 1997). The optimization of DCS model is a search problem in multidimensionnll space, where each solution makes a potential probability correction scheme. Next, we uncover the overall framework that apply SA to solve our model.
3.1 Overall Framework
The overall solution framework for DCS is depicted in Fig. 2. The combinatorial optimiza- tion relies on labeled set of these initial class probabilities and their ground-truth labels. The labeled set can be simply taken from the full or a subset the classification task's training set. After ICL output class probabilities are obtained, the optimization process starts with an initial solution. The iteration repeats until either a minimum temperature is reached maximum number of iterations is reached, and obtain the optimal correction functions. In actual inference, the selected functions directly rectify a test instance's ICL output class probabilities and result in corrected predictions.

---

Published as a conference paper at COLM 2025
3.2 Initial Solution
We can keep the initial probabilities unchanged as the initial solution for the DCS math model. The fuzzy rule set Membershhi [ designed in this paper must include Don't Change, the triangular membership function with dg 0, by cg the original probability unchanged. solution can be chosen as: ; = Ko, for class j = 1,...,,, ,.. For SA, the initial solution does not affect the quality of the final solution, but influences the annealing process, and we empirically start with unchanged probabilities (Kirkpatrick et al., 1983).
which will keep the corresponding selection index is kg, the initial
3.3 Neighborhood Solution
It is necessary to generate new solution in the neighborhood of the current solution. If .,.,,,,, N,, we can randomly select a class, for example, j' whose current solution is ły ky, and perturb it by sampling kAlew from the set D ky as the new value for the variable Ejr. That is, ły 7 -Nlow In this way, we obtain neighboring solution: gNew -š1 ky, K-New,...,, .....,N ky). The transition from the current solution at a given temperature to the new solution is a Markov chain (Kirkpatrick et al., 1983; Cerny, 1985), and the transition probability is determined by the Metropolis criterion Paccept = e AZ/T the new solution better (AZ < 0), accept the new solution; if the new solution is worse (AZ 2 0) and uniform random number r ~ U(0,1) is smaller than Paccept, namely < Paccept, the new solution is also accepted.
the current solution čj kj, for class) j 1,...,
3.4 Annealing Schedule
An annealing schedule is characterized by the initial value of the temperature, the cooling rate, the inner-loop criterion, and the outer-loop criterion.
The initial temperature is chosen that the acceptance probability is relatively close to one. The higher the initial temperature, the more possible to obtain a high- quality solution, but the longer the computation time. In this work, the initial temperature is set to 200,000.
Initial Temperature:
Inner-Loop Criterion: The inner loop simulates the process of the system reaching equilib- rium at a given temperature. Theoretically, there should be a sufficient number of iterations. The more decision variables there are, the more iterations the inner loop should have. There- fore, in this paper, the inner loop stops when either the number of accepted solutions reaches A times of the number of variables, or the number of solutions generated exceeds A2 times of the number of variables. The value of A2 are tuned alongside other hyperparameters.
Outer-Loop Criterion and Cooling Rate: Generally, the termination temperature should be set to a sufficiently small positive number. The temperature update function follows geometric rate: Teurr+1 & Teure, where set as 0.95 in this paper.
4 Experiments
We assess the capabilities of DCS in reducing class accuracy differences and enhancing overall accuracy with three ICL settings: 1-shot, 5-shot, and N-shot. Demonstrations in the prompt are randomly selected from the optimization set for 1-shot and 5-shot cases, while the N-shot demonstration cascades examples from each class, to minimize under- representations of any class. DCS is compared against SoTA methods, DNIP (Lin & You, 2024a) and FuRud (Lin & You, 2024b). We prompt LLMs with an A100 GPU to obtain initial ICL outputs, following DNIP's hyperparameters, then we run simulated annealing on CPUs. Evaluations are performed on seven multi-class text classification tasks across general and biomedical domains, including AGNews (AGN, A-class;(Zhang et al., 2015)), DBpedia (DBP, 14-class/Auer et al., 2007)), SST-5 (SST, 5-class/(Socher et al., 2013)), TREC (Voorhes & Tice, 2000; Li & Roth, 2002)), RTE (Dinary:(Dagan et al., 2006)), DDI

---

Published as a conference paper at COLM 2025
(5-clas;;Segguaa-Bers Bedmar et al., 2013)), and PubMedQA (PMQA, 3-class;([in et al., 2019)). Appendix D presents more details.
4.1
Main Results
Setup We perform quantitative evaluations on Llama-2-13B, for its representativeness widely applied open-source LLM. Indeed, Llama-2-130 not particularly big in parameter size, but its architecture can be seen basic building blocks for many advanced LLMs (Chiang et al., 2023; Rozière et al., 2024; Liu et al., 2023), and we expect similar findings in larger model variants models similar in architecture, which will be discussed in Section 4.3. Table reports our main findings.
Acc
COBias
Method
General Doeaatn
Biomedical Domain
General Domai
Blomedical Domain
Avg JAGN
TREC
Avg
DDI
Aug
AGN
TREC
Aug
PMQA
T-stot
ICL
posl
ICL
9944
70.
79.9g
88.2.a
469
685ua
71.
312
7:2os
56.329
4405
35.4
28.3
16.2,
53.1a
35.9s
43.4gp
53.4
45.69
61.2s
ADNIP
87.8
143
142,,
FulRdd
86.
92.7.a
5000
7844
745.a
44.0
20
38.641
151
10.9
88
362aa
135
73
19.5
11
272
+DCS (Churs
722
77.
88.2
94.727
502u
81.5gs
74.
56.1
52.9
63.3g
150
10.7
6.2
6.0ba
23.7
128
5.0
25.8
164u3
35.2na
Swhett
dehiniing
93
139.
24
DDNPP
71.9
77.9
88.5cz
95.8
52.9s
78.8s
54.1aa
39.9ta
1033
70k
571
158
1444
3.
13.1
1655
9.60
+ FuuRud
70.6
77.9
67.8
95.5
53.
76.8p
75.8
52.5
60.3
64.3ss
14.7
11.8
10.0
576
22.1es
19.0
1.0gs
22.1
19.3s
25.1ms
DDS (Cums)
7488
79.0
88.8ga
96-3aa
54.2go
79.7.
76.143
64.3
57:143
21.4g
14.7
777
67
47a
11.763
14.0g3
1.6g
30.2
45.4
N-shor IC2
post hor dehioning
ICL
61.
712
83.5
95.212
50.3s
67.0
75.0s
31.0
971
32.3a3
22.6
23.8
14.9
7.0g2
36.51
38.6
225ra
30.3
39.75
20.4
DDNIP
71.7
79.2
M8.70
96.63
5133
82774
76.777
49.5
63,6
58.324
75
63
7.3
433g
28u
121
5.0
10.6
12.5p
8.7
PuRud
87.8k
78330
DCS (Ours)
88.9.
96.8.
542
834a
767
62.4.s
60.3a
74
16%
11.
50
23.3
28
17.
Table 1: Main results
The proposed method achieves SoTA overall accuracy improvements on all tasks except for RTE across all ICL settings. As for RTE, due to its simplicity as a binary classification task, all three methods obtain similar accuracy and COBias improvements. Meanwhile, achieves SoTA COBias performance on general domain tasks. In general, its COBias is on par with the current best methods, with average COBias over all seven tasks falling between that of DNIP and FuRud. On AGNews, DBpeida, and TREC, the proposed method achieves both more overall accuracy improvements and more COBias reduction than DNIP and FuRud across all ICL settings, demonstrating the superior performance gains achieved by ensembling class-level and sample-level debiasing. Weak classes are elevated. We show in Section 4.2 that DCS powers up the low-accuracy classes. We observe that weaker classes are often corrected by rules, while stronger classes are most likely corrected by weights, further highlighting the need of an ensemble method.
Annealing Time
Figure 3 plots the time used to optimize correction functions (left axis) and the search space size (right axis) for each task. For fair comparisons, we fix the weight scale for weight correction to 30 alongside 19
100
Average time finst opt mistionn overthree runo of ouputs (diffennnt demonetmbong
700
600
triangular membership functions, making total of 49 possible correction functions and 49 N possible
Annesling time/sec.
/sec
Search space size
500
60
59.7
58.1
assignments across all variables. DCS is computa- tionally efficient where all tasks finish optimization within 150 outer loops, ranging from a maximum of 96.6 seconds to minimum of 5.4 seconds. Notably, SST-5, DDI, and AGNews have a relatively longer time due to having a larger optimization set (Ap- pendix D). In general, although larger optimization sets and search spaces need more time to converge, the annealing steps needed for a task can be further tuned to achieve cost-effective gains on downstream applications.
400
1as
300
294
20-
200
Search
12.3
M(D.+Dpl
5% 92-
100
TREC
SST
PMQA RTC
Figure 3: Task annealing time
7

---

Published as a conference paper at COLM 2025
4.2 Overall Probability Correction Scheme
The overall probability correction scheme demonstrates the ratio of class-level corrections to sample-level corrections among all classes. Which type of correction should be used for class is automatically determined by our method. Recall that: Class-level correction: Prob- abilities of the same output class are corrected uniformly using a same correction weight across all samples. It operates at the class level, not accounting for sample-based differences. Sample-level correction: This a more refined correction using membership correction func- tions. Instead of a single weight, a unique mapping function is applied for each class. This allows varying degrees of correction across different samples some probabilities may be adjusted significantly, others minimally, and some may remain unchanged capturing the nuances of individual samples, leading to more precise adjustments. By combining them, our method ensures balanced and adaptive correction, enhancing performances.
Classes that use weight corrections
Classes that use membership corrections
Ratio memberhsip currections weight corrections
#Classes that membership corrections in seed0/1 20.00/11/22 run
Remarks any weakest class use membership corrections
Dataset
Classes
AGN
3.7
0.3
0.08
0/1/0
The weekest class Sports from seedl run.
Acc: 0.75 (ICL) 0.98 (currectes))
12.3
2/2/1
weskest class Ternible
run,
Acc: 0.21 (ICL) -> 0.54 (corrected)
SST
4.3
0.7
0.16
1/0/1
The weakest class Terrible from seed2 run,
Auc: 0.09 (CCL) 0.64 (coerected)
TREC
5
0/2/1
Acc: 0.04 (ICI. (176 (corrected)
Enfity
rum,
RTE
0.7
1.3
1.86
1/2/1
The weakest class Folie from seed1 run,
Acc: 01.4 (ICL) > 0.28 (cornected)
The weakest class Negatize from seed0
Acc:
0.73 (conected)
weakest class Negatize from seed run,
DDI
2.7
2.3
0.85
3/2/2
Acc: 0 (ICL. 0.46 (corrected)
The weakest class Negatice from seed2 run.
Auc: 0.02 (ICL) 0.93 (coerected) rum,
class Yes The Acc: 0.38 (ICI.) 0.62 (cornccted)
PMQA
2.7
0/1/00
Table 2: Overall probability correction scheme (N-shot)
We exemplify the overall probability correction scheme with the N-shot case in Table 2. Numbers of classes are averaged over three runs of different demonstrations. As expected, most classes select class-level weight corrections. Membership corrections are less selected, but kest classes benefit greatly from them, where 6 out of tasks have selected mem- bership corrections to correct their weakest classes (at least once over three runs). This demonstrates the necessity of enabling sample-level corrections to boost the performance of weak classes, and letting the optimization decide if sample-level corrections are needed.
4.3 Larger Models on DDI (Biomedical-domain)
Similar to its smaller variant Llama-2-13B, 1-shot ICL on Llama-2-70B also manifests class accuracy imbalances. Especially on the biomedical domain DDI classification, us- ing 1-shot prompting, Llama-2-13B's aver- age ICL accuracy on DDI is 7.2% with CO- Bias 45.6% shown in Table 1, while Llama- 2-70B obtains 8.8% accuracy with COBias 36.0% as shown in Table 3, showing that the larger variant is still highly prone to biased predictions on biomedical tasks. This sug- gests that the class accuracy imbalance may not be effectively alleviated by increasing model sizes, and the proposed post-hoc cor- rections can exactly help with the situation. Due to effectively correcting probabilities for
Model
Acc
Prompt Demon.
Neg.
Hil.
Class
Mech.
Ade:
Inl.
Liome-2-130
seedü
0.0
100.0
0.0
2.3
ICL
7.2
45.6
sced1
91.4
17.7
39.6
seed2
54.8
88.5
+DCS
52,9
16.4
sced0
38.1
25,0
52.3
31.3
seed2
11.5
Zlome-2-700
4.1
19.8
ICL
888
56.3
58.3
+DCS
63.7
23.6
35.6
47.9
seed2
80.5
16.1
18.6
33.3
Table 3: Significant DDI Performance gains on both Llama-2-13B and Llama-2-70B.
8

---

Published as a conference paper at COLM 2025
low-accuracy classes shown by Table 3, our method demonstrates significant accuracy enhancements on DDI, across both small and large model variants. For more Llama-2-70B performance gains, detailed scores can be found in Appendix E.
4.4 Ablations
Some may contend that more overall accuracy enhancements can be compensated by less COBias reduction. While tradeoffs between overall accuracy and COBias generally exist, relaxing COBias reduction (resulting in larger COBias score) can not guarantee higher
overall accuracy (Table 4), showing the need of dedicated ZCOBias in the objective function.
AGN
DBP
SST
TREC
RTE
DDI
PMQA
Prompt
Obj.
Acc.
CB
Acc.
CB.
Acc.
CB.
Acc.
Acc.
CB.
Acc.
CB.
Acc.
CB.
Full
88.2
6.2
94.7msa)
6.0
50.2
23.7
81,5pm2
(2.8
74.4
5.0
52.9
16.4
63.3
35.2
1-shot
2
88.4
7.8
92 8/42)
8.6u74
5144
50.9
80.6e120
16.7umz
75.7
12.8
82.2
39.8
71.3
58.8
2 PM
88.5
7.5
Iina
7.5uxn
51.7
81.1mm2
15.1ua
75.8
13.0
82.9
40.0
71.1
59.1
Full
68.8
6.7
96.3mm
4.7us
54.2
11.7
79.7
14.0
76.1
1.6
57.1
21.0
71.4
43.4
5-shot
Zfr
88.9
7.8
95. 0m14)
7.lum
54.2
41.6
79 Scaaa
14.6uax0
76.2
13.7
82.8
40.0
76.7
58.8
PMM
7.9
95.9
5.5gam
54.9
78, 78.9
21.0gas.
76.9
10.2
76.8
58.8
Full
88.9
7.1
96.8
4.1g28
54.2
16.9
83.4ess
11.16m7
76.7
5.0
62.4
28.6
60.3
17.9
N-shot
2
98.8
7.5
96.3m1
5.1uss
54.4
37.4
81.9
18. 18.4um00
76.9
17.0
82.8
40.1
76.0
60.5
2 1MM
7.5
96.3m1
4.8um
83.3
15.3gm
77.5
15.7
76.1
60.7
Table Z objective ablations; average scores over three random seeds are reported. Improvements over ICL scores are shown by (1) and (L).
To see this, we ablate the overall error term Z from the objective function and optimize solely for Frr, On DBpedia and TREC, DCS with only Zir did not obtain a higher accuracy than the DCS with the full objective function. This is because the optimization without an explicit ZCOBias term may favor higher-accuracy classes which are easier to improve, and becomes "lazy" to improve classes that have low accuracies initially. Applying the learned correction functions will result in a good overall test accuracy with fair COBias. On the other hand, explicitly setting ZCBiis in the objective function can help enforce fair improvements for all classes while optimizing overall accuracy. It could bring more accuracy gains for lower-accuracy classes, and thus obtaining even higher overall test accuracy with lower COBias in some cases, especially when there are only few low-accuracy classes among many higher-accuracy classes initially, like DBpedia and TREC. This suggests that balancing class accuracies can help with overall accuracy gains. Additionally, we ablate both accuracy objectives Err and PMII and optimize for these two terms without ZCOBias, On DBpedia and TREC, its performance generally falls between optimizing for Z and the full objective, further suggesting the importance of COBias in reaching optimal solutions.
4.5 Discussions
We visualize the ablation of sample-level and class-level rebalancing in Appendix F, demonstrating that sample-level rebalancing significantly elevates weak classes, while class- level rebalancing steadily balances the stronger ones. Moreover, regarding more model families, we provide experimental results with Gemma-2-2B in Appendix G, which obtain consistent improvements similar to the Llama-2 cases, demonstrating that our method is applicable to LLMs of varied sizes and families. Lastly, with respect to the time and cost of applying the DCS method in real-world scenarios, we elaborate below.
Accessibility: DCS is model-agnostic, requiring only output probabilities, making it particu- larly suitable for large or closed LLMs where only / probabilities are accessible.
Cost of post-hoc optimization: The post-hoc, offline optimization of DCS requires no model architecture modification and prompt engineering. The annealing time averages 46 seconds (computed from Figure 3). Hyperparameter tuning can be done efficiently because most hyperparameters are pre-configured in practice, with only a few scalar values (e.g., s, r) requiring tuning. Moreover, since the class-level correction method DNIP demonstrates low-

---

Published as a conference paper at COLM 2025
data optimization capability, we'd infer that DCS is amenable to low-data scenarios. These make DCS particularly valuable in specialized applications where fairness and accuracy are paramount.
Negligible inference overhead: The learned correction functions are reused on-the-fly with mere milliseconds in prediction, introducing virtually no computation overhead or latency.
Scalability: Our method exhibits linear scaling with the number of classes (N). At each tem- perature in simulated annealing, the number of searches remains within several multiples of the search space size N(Dp + Dw), ensuring practical feasibility across small and large classification tasks.
Conclusion and Limitations
This paper proposes DCS, a post-hoc ensemble debiasing framework to correct ICL probabil- ities across class and sample level. It boosts low-accuracy classes, while keeping the strong ones. It achieves SoTA accuracy on seven benchmark text classification tasks, while effec- tively mitigating class performance imbalances. The resulting probability correction scheme clearly shows that combining membership correction and weight correction is essential to elevate weak classes. For limitations and future work, the post-hoc correction framework can be extended to other useful scenarios, such as text generation. Combining DCS into the decoding process can help rectify the probabilities of top candidates for the first token (or first several tokens) to lead better, less biased response. More future directions are discussed Appendix H.
Acknowledgments
Yang You's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ByteDance grant, ARCTIC grant, SMI grant (WBS number: A-8001104-00-00) and Alibaba grant.
References
Tobias Achterberg. SCIP: Solving constraint integer programs. Mathematical Program- ming Computation, 1(1):1-41, 2009. doi: 10.1007 (0.1007//s2233-000-000000 URL https: //tink// http:/ger..oo//rrllee/eeeee /tt.:/aacce.eee.100////255222220
0001-1.
Akshay Agrawal, Robin Verschueren, Steven Diamond, and Stephen Boyd. A rewriting 2018.
system for convex optimization problems. Journal of Control and Decision,
Sören Auer, Christian Bizer, Georgi Kobilarov, Jens Lehmann, Richard Cyganiak, and Zachary Ives. DBpedia: A Nucleus for A Web of Open Data. In Proceedings of the 6th International The Semantic Web and 2nd Asian Conference on Asian Semantic Web Conference, pp. 722-735, 2007.
Jonathan Berant, Vivek Srikumar, Pei-Chun Chen, Abby Vander Linden, Brittany Harding, Brad Huang, Peter Clark, and Christopher D. Manning. Modeling biological processes for reading comprehension. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1499-1510, Doha, Qatar, October 2014. Association for Computational Linguistis.. doi: 201111/111D1411111 URL D14-1159/.
org/
Alberto Mario Ceballos-Arroyo Monica Munnangi, Jiuding Sun, Karen Zhang, Jered McIn- erney, Byron C. Wallace, and Silvio Amir. Open (clinical) LLMs are sensitive to in- struction phrasings. In Dina Demner-Fushman, Sophia Ananiadou, Makoto Miwa, Kirk Roberts, and Junichi Tsujii (eds.), Proceedings of the 23rd Workshop on Biomedi- cal Natural Language Processing, pp. 50-71, Bangkok, Thailand, August 2024. Associ- ation for Computational Linguistics. doi: 10.18653 1006553/112224.iom/ppp // //ac..anthology.org////oo bitee-iiooll..//
URL https:
10

---

Published as a conference paper at COLM 2025
V. Cerny. Thermodynamical approach to the traveling salesman problem: An efficient simulation algorithm. Journal of Optimization Theory and Applications, 45:41-51, 1985. URL
https://doi. org/10///////// 10/./////00
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric Xing. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality, March 2023. URL
20t/:2222330---immmm vicunu....
Ido Dagan, Oren Glickman, and Bernardo Magnini. The PASCAL Recognising Textual Entailment Challenge. In Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Tectual Entailment, pp. 177-190, 2006.
Parag Pravin Dakle, Serdar Kadioglu, Karthik Uppuluri, Regina Politi, Preethi Raghavan, SaiKrishna Rallabandi, and Ravisutha Srinivasamurthy. Ner4Opt: Named entity recogni- tion for optimization modelling from natural language. In CPAJOR, pp. 299-319, 2023. URL http::::/doooor////////
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computaatinna Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653 (00663/vl//// 11118653/11/N1111140 URL https: http::///caaaahlooooooooo Nrg/NN1-1423/.
Steven Diamond and Stephen Boyd. CVXPY: A Python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-55, 2016.
Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Nicholas Joseph, Ben Mannt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. A mathematical framework for transformer circuits. Trans- former Circuits Thread, 2021. URL https: tttt:::://tranffomme---oo cir rcuits. pub/ gub/2021// /framework index. html.
Atticus Geiger, Hanson Lu, Thomas F Icard, and Christopher Potts. Causal abstractions of neural networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https://openrevieww net/ forum?id-=muuDttDDnG
Dan Goldwasser, Vivek Srikumar, and Dan Roth. Predicting structures in NLP: Constrained conditional models and integer linear programming in NLP. In Tutorial Abstracts at the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Montréal, Canada, June 2012. Association for Computational Linguistics. htt.tt::/aallaatholloooo org/N12-4400//
Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2024. URL httpps://ww. gurobi, com.
Raymond Hemmecke, Matthias Köppe, Jon Lee, and Robert Weismantel. Nonlinear Integer Programmiing pp. 561 561-618. Springer Berlin Heidelberg, Berlin, Heidelberg, 2010. ISBN 978-3-540-68279-0. doi: 170-0077777783-5--6777777 978-3-540-68279-0.10
URL https:://oi..orrg//// 1007/
Kashif Hussain, Mohd Najib Mohd Salleh, Shi Cheng, and Yuhui Shi. Metaheuristic re- search: A comprehensive survey. Artificial Intelligence Review, 52(4))2191-2233, 2019. doi: 10.1007//s00422-0179966555 s18462-917-9605-2.
URL httpps://ling www.iiik.ssiiigr..oom sprringer.com/artiil//t1 1007/
11

---

Published as a conference paper at COLM 2025
Andoni I. Garmendia, Josu Ceberio, and Alexander Mendiburu. Applicability of neural combinatorial optimization: A critical view. ACM Trans. Evol. Learn. Optim., 4(3), July 2024. doi: 00.1145/364765 URL httpps://oi:: org/10. 1145/3647644.
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. Certified robustness to adversarial word substitutions. In Proceedings of the 2019 Conference on Empirical Meth- ods in Natural Language Processing and the 9lht International Joint Conference on Natural Language Processing (EMNLP-IJCNLP?) pp. 4129-4142, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18653/ 10.18633/v1/D19-1221 v1/D19-1423. URL httpps://clie https://aclanthooog..or////11/1
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. PubMedQA: A Dataset for Biomedical Research Question Answering. In Proceedings of the 2019 Conference
on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2567-2577, 2019. URL https:// aclanthology wwwggy...rgggo. org/D1g 10119-1255
Michael Jünger, Thomas M. Liebling, Denis Naddef, George L. Nemhauser, William R. Pulleyblank, Gerhard Reinelt, Giovanni Rinaldi, and Laurence A. Wolsey (eds.). 50 Years of Integer Programming 1958-2008: From the Early Years to the State-of-the-Art. Springer, 2010. ISBN 978-3-540-6827455 doi: 10.1007 /977-3-350--62279-00 URL https:::://lnk springge....ssing.r......
4
com/book/10 000007997783-44-6277777
S. Kirkpatrick, C. D. Gelatt, and M. P. Vecchi. Optimization by Simulated Annealing. Science, pp. 671-680, 1983. URL httph:ttp/////tt //www wwwwww..ccccceeeeeeccccce.. wtrg/doi/abs//1/ 1126/science 4598. 671.
Terry Koo, Alexander M. Rush, Michael Collins, Tommi Jaakkola, and David Sontag. Dual decomposition for parsing with non-projective head automata. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 1288- 1298, Cambridge, MA, October 2010. Association for Computational Linguistics. URL https://aclanthoooog.ooom -t1g:///00...gg1/-115555.
A. H. Land and A. G. Doig. An automatic method of solving discrete programming problems. Ecomometrica,, 28(3):497-520, 1960. ISSN 00129682, 14680262. URL http:
// h/www.jjtoot www..jstor.org/stable//eeoo org/stable//110013
Kenneth Li, Oam Patel, Fernanda Viégas, Hanspeter Pfister, and Martin Wattenberg. Inference-time intervention: Eliciting truthful answers from language model. In shirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
Xin Li and Dan Roth. Learning Question Classifiers. In COLING 2002: The 19th International org/ 0rg5002-1150
Conference on Computational Linguistics, 2002. URL
Ruixi Lin and Hwee Tou Ng. System combination for grammatical error correction based on integer programming. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pp. 824-829, Held Online, September 2021. INCOMA Ltd. URL ht http://ccaaaattooo...oooo wwwp//...org///// 2021. ranlp-1 1.94/.
Ruixi Lin and Hwee Tou Ng. Mind the biases: Quantifying cognitive biases in language model prompting. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 5269-5281, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653 06633/v12222. findings-acl.324. URL https::/aclantthoo.o org/ findings--aaai acl 324/.
Ruixi Lin and Yang You. COBias and Debias: Minimizing Language Model Pairwise Accuracy Bias via Nonlinear Integer Programming, 2024a. URL https://arxiv.or 2405.07623.
Ruixi Lin and Yang You. Let the Fuzzy Rule Speak: Enhancing In-oontext Learning Debiasing with Interpretability, 2024b. URL. https: httt::::/aaaii.oog/a////2222 19018.
12

---

Published as a conference paper at COLM 2025
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: net
// www.eeteooruumiddddeeeex.l...
David G. Luenberger and Yinyu Ye. Linear and Nonlinear Programming, volume 116 of International Series in Operations Research & Management Science. Springer, 3rd edition, 2008. ISBN 378-0-387-74503-4 doi: 10.1007 (978-0-387-74503-- URL httpp::/linkk springer.. com/book/10. 0. 1007/9755 0-77788---877550----
André Martins, Noah Smith, and Eric Xing. Concise integer linear programming formu- lations for dependency parsing- In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joiní Conference Natural Language Processing of the AFNLP, pp. 342-350, Suntec, Singapore, August 2009. Association for Computational Linguistics. URL https: /aac::/aclllaa www.aaaannnnotooooooooo ooo.aaatoooo...or//00-1..3..
George Nemhauser and Laurence Wolsey. The Scope of Integer and Combinatorial Optimization, chapter I.1, pp. 1-26. John Wiley & Sons, Ltd, 1988. ISBN 9781118627372. doi: https: //doi.org/ //doi.orgg/1..002///7///3 /0011002//71111177777.ttt URL https://oollimelibraay wwley..ww....................... waally..cco/// com/coooooooooooo abs/10.
Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. In-context learning and induction heads, 2022. URL nttps://arxinn www.aaiwwi..rgga...a../eaereeaae org/abs/2209. 11895.
Dan Roth and Vivek Srikumar. Integer linear programming formulations in natural language processing. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Tutorial Abstracts, Valencia, Spain, April 2017. Association for Computational Linguistics. URL
Dan Roth and Wen-tau Yih. A linear programming formulation for global inference in natural language tasks. In Proceedings of the Eighth Conference on Computational Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pp. 1-8, Boston, Massachusetts, USA, May 6 May 2004. Association for Computational Linguistics. URL https: //aclanthology hwwptthhooogyooog/wweeeeeee wwwp/04-22/000 2401/.
Dan Roth and Wen-tau Yih. Integer linear programming inference for conditional random fields. In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, pp. 736-743, New York, NY, USA, 2005. Association for Computing Machinery. ISBN 1595931805. doi: 10.1145 10.145/110235..102444 1102444.
URL https://doi. 1145//1111111/1112551111111
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024. URL https: httpp::////xxxv..rrr// /b3// 12950.
Alexander M. Rush, David Sontag, Michael Collins, and Tommi Jaakkola. On dual de- composition and linear programming relaxations for natural language processing. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pp. 1-11, Cambridge, MA, October 2010. Association for Computational Linguistics. URL https///accaanttoooooooo/o//////
Linus Schrage. Optimization Modeling with LINGO. LINDO Systems Inc., Chicago, II., 5th edition, 1999.
Melanie Sclar, Yejin Choi, Yulia Tsvetkov, and Alane Suhr. Quantifying language models' sensitivity to spurious features in prompt design or: How learned to start worrying
13

---

Published as a conference paper at COLM 2025
about prompt formatting. In The Twelfth International Conference on Learning Representations, 2024. URL htttt::://peeeeviewe wttpeerreiieeeeeeeoorrmmea
Isabel Segura- -Brmmr,, Paloma Martínez, and María Herrero-Zazo. SemEval-2013 Task 9
: Extraction of Drug-Drug Interactions from Biomedical Texts (DDIExtraction 2013). In Second Joint Conference Lexical and Computational Semantics (*SEM), Volume 2: Proceedings of the Seventh International Workshop on Semantic Evaluation (SemEval 2013), pp- 341-350, 2013. URL https: / /aaaaaaaaannnooooooe org/ 200--22566.000 pdff.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over Sentiment Treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, 2013. URL https://aclanthone
org/D13-1170.
Vivek Srikumar and Dan Roth. The integer linear programming inference cookbook, 2023.
URL https://arxiv.. org/ abs//30/./22/.. 00171.
Michael Steinbrunn, Guido Moerkotte, and Alfons Kemper. Heuristic and randomized optimization for the join ordering problem. The VLDB Journal, 6:191-208, 1997. URL https://doi, org/10. 10077
Christos Thrampoulidis. Implicit optimization bias of next-token prediction in linear models. In Advances in Neural Information Processing Systems, volume 37, pp. 22624- 22656, 2024. URL
fiaaper.iiles/pppr//2///// pdf.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open Foundation and Lune-Tuned Chat Models. arXiv preprint ar Xiv:2307 2tt.:2307..0088,,, 2023.
Jesse Vig, Sebastian Gehrmann, Yonatan Belinkov, Sharon Qian, Daniel Nevo, Yaron Singer, and Stuart Shieber. Investigating gender bias in language models using causal media- tion analysis. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 12388-12401. Curran Associates, Inc., 2020. URL httpp:::ppoocceeot////o/ hewprrrreeeeeennnnnnniierrisreee httpppssccpppp..ppe wttpeefffile//pap/////// file 7ii2-25500e227775551226622222222
pdf.
Ellen M. Voorhees and Dawn M. Tice. Building question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, pp. 200-207, 2000. URL https: //dp::://oi.. org/10. 11100.1..11. 1145/345508. 345577.
Ming- Wei Wang, Nicholas Rizzolo, and Dan Roth. Integer linear programming in NLP- constrained conditional models. In NAACL HLT 2010 Tutorial Abstracts, pp. 9-14, Los Angeles, California, June 2010. Association for Computational Linguistics. URL https: //aclanthology. org 0N0500
Xuezhi Wang and Denny Zhou. Chain-of-thought reasoning without prompting. In The Thirty-eighth Anmull Conference on Neural Information Processing Systems, 2024. URL https: //openreview wwtpeeewwwweeeeeorrrddd//------e
14

---

Published as a conference paper at COLM 2025
Yuanbin Wu and Hwee Tou Ng. Grammatical error correction using integer linear pro- gramming. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1456-1465, Sofia, Bulgaria, August 2013. Associa- tion for Computational Linguistics. URL
org/P13-1113/
Haoting Zhang, Jinghai He, Jingxu Xu, Jingshen Wang, and Zeyu Zheng. Enhancing language model with both human and artificial intelligence feedback data. In 2024 Winter Simulation Conference (WSC), pp. 299-310, 2024. doi:
10838929..
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Net- works for Text Classificatton. In Advances in Neural Information Processing Sys- tems, 2015. URL https://prroceeding: hww.:/ceeeddinn.nnnniiiii..... neurips. httpscccppppe//iieee .ddf.
Yize Zhao, Tina Behnia, Vala Vakilian, and Christos Thrampoulidis. Implicit geometry of next-token prediction: From language sparsity patterns to model representations. In First Conference on Language Modeling, 2024. URL https://pperreiiww.eee.. qyilOnIRHI.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate Before Use: Improving Few-shot Performance of Language Models. In Proceedings of the 38th International Conference on Machine Learning, pp. 12697-12706, 2021. URL https: //proceedings. www.pprees//3///haaaaae html.
Han Zhou, Xingchen Wan, Lev Proleev, Diana Mincu, Jilin Chen, Katherine A Heller, and Subhrajit Roy. Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineering. In The Twelfth International Conference on Learning Representations, 2024. URL https: http///ppppeeeeeeee/eeeeeeeeeeet www.ee/ooorrmmdd-33FF
A Discussion on Solution Methods for NIP
Typical solutions for combinatorial optimization problems include the branch-and-bouu (B&B) algorithm, which is useful for solving mixed-integer linear programs (Land & Doig, 1960). Solvers are made into both open-source software such as CVXPY (Diamond Boyd, 2016; Agrawal et al., 2018) or SCIP (Achterberg, 2009), and commercial solvers like Gurobi (Gurobi Optimization, LLC, 2024) or LINGO (Schrage, 1999).
We need to point out that B&B and other related algorithms are not very suitable to search solutions for the proposed nonlinear integer programming based optimization model, which contain discontinuous functions and is inherently non-differentiable Relaxation to subproblems is needed if B&B has to be used. Relaxation can be done by creating Lagrange multipliers and Lagrangian functions for each constraint, which is often easier to solve than the original nonlinear integer programming problem, because only continuous variables are involved (Luenberger & Ye, 2008; Jünger et al., 2010). The relaxed problem finds the optimal value of the Lagrangian function, which is a lower bound for the original problem, and it used as bound in the B&B search. However, solving the subproblems efficiently poses challenges, as B&B is generally not polynomial-time algorithm.
Instead, we can opt for metaheuristics (Hussain et al., 2019), such simulated annealing (Kirkpatrick et al., 1983; Cerny, 1985), to find the optimal solution. Although heuristic search can get slower with increasing search space size (Jia et al., 2019), its computational complexity is within polynomial time (Lin & You, 2024a). To enable application of our proposed method to tasks with even more classes, we adopt simulated annealing (SA) to solve our model.
In SA, we define an initial temperature and an annealing schedule including the cooling rate and the stopping criteria. In each iteration, neighborhood solution is generated by perturbing the current solution. The algorithm calculates the objective function value for the new solution (Step Evaluate Z in Fig. 2). If the new solution is better, accept it; if the new solution is worse, accept it with probability determined by the Metropolis criterion. Then the temperature is reduced according to the cooling rate, until convergence. In addition, the
15

---

Published as a conference paper at COLM 2025
initial solution can also be randomly generated, where N numbers are randomly selected from (I,..., Dr + Dw) } as the initial values for the selection variables čj's.
B List of notations
The parameters, sets and decision variables used throughout this paper are listed in Table 5,
Symbol
Description
Parameters
Pm
The ICL output token's class probabilities of the m-th instance in a dataset
Wk
The k-th weight coefficient where k The k-th membership function where k (Dr + 1,.., DF + Dw]
Hx
Sets
I Weight
The set of Dw weight coefficients, equally spaced discrete values from 0 to 1
I Membership
The set of Dy triangular membership functions The set of selection indices
DD
Variables
Mmu 9m
The label and prediction of the m-th instance The integer selection variable for class
Table List of notations
C Details on the Objective Function
Em COBBia PMI respectively follows from the three terms of Eq. 5 in (Lin & You, 2024a),
which include:
The overall error rate for M optimization instances:
1 M
L
Σ
1[9m= ymb, where Im arg max P'mn I,..,,M
(6)
nE(1,..,N)
COBias of all class pairs:
Z
COBlas
N
-IN-1 Σ
Σ
Ag Agp. where Ace is class c*'s accuracy
(7)
PMI between ground-truth truth instances of class j and predictions of class j:
N
PMI
Σ
PMI
(8)
where
f($), 5)) f($i)
PMI: PMI log
for j = L,..,,N
(9)
Here, Ši and Si denote the instances of prediction j and true label respectively, f()) is the ratio between the number of instances with prediction j and the total number of instances, similarly, f(i) the ratio between the number of instances labeled j and the total number of instances, f f(ii, Si) is the ratio between the number of correct predictions of class and the total number of instances.
16

---

Published as a conference paper at COLM 2025
D Experimental Setups
For each evaluation task, we tune the weight scale for weight correction, & and used in the objective function, and the simulated annealing algorithm's 's number of outer loops, inner loops, the initial temperature, and stopping criteria (stopping temperature, the maximal number of solutions generated, and the maximal number of accepted solutions) on devel- opment set. The optimization/ /evelopment set split is 0.95/0.05. Dataset preprocessing follows DNIP (Lin & You, 2024a), and their statistics are shown in Table 6.
Optimization set size (inel. dev)
Evaluation set size
Dataset
Prompt
Please classify the news articles into the categories of World,Sports, Business, and Technology. \n)n
AGN
10,000
5,000
Article: [Input]
Answer:
Please classify the documents based on whether they are
about a Company, School, Artist, Athlete, Politician, Transportation, Building, Nature, Village, Animal, Plant, Album, Film, or Boo
DBP
10,000
5,000
Article: [Input]
Answer:
Review: [Input]
SST
8,544
2,210
Sentiment:
Please classify the questions based on whether their answer type is Number, Location, Person, Description, Entity, or Abbreviation. \n),
TREC
5,452
500
Question: [Input]
Answer Type:
[Input premise] question:
RTE
2,490
277
[Input hypothesis]
True or False?
answer:
Please choose most suitable answer from Negative, Effect, Mechanism, Advice, Interaction, for the drug-drug interaction relation between the @drugs pair in the following description.)ni
DDI
10,000
5,716
Description: [Input]
Answer:
Please choose a most suitable answer from yes, no, or maybe, for the following question given context. inin
PMQA
1,000
500
Context: [Input context]
Question: [Input question] yes, maybe?
Answer:
Table 6: Dataset statistics
E Full Performance Comparisons on Llama-2-70B
Our method effectively applies larger LLM variants, showcased by Llama-2-70B. Debi- asing results using DCS on 1-shot ICL outputs in Table show consistent improvements in average overall accuracy across three seeds demonstrations, outperforming DNIP and FuRud. Our method achieves better COBias reduction than the other two methods on general domain tasks, and better accuracy with comparable COBias reduction on biomedical domain tasks. The takeaway is, increasing LLM sizes without post-hoc corrections may not always be an effective way to resolve output class accuracy imbalances, which further validates the necessity of the proposed post-hoc correction method.
17

---

Published as a conference paper at COLM 2025
Acc
COBias
Method
General
Biomedical Domain
General
Biomedical
All
AGN
DBP
SST
TREC
Avg.
DDI
PMRA
All
AGI
TREC
Aug)
DIN
PMQA
I-shut
ICL
past-hoe defaasig
ICT.
66.4
762
87
94.1g
41&
79.5gx
78
41.7
886
74.5gs
31.8
25.1
14.5%
8312
547;4
224,a
255ps
4888
36.02
61.5p
89.8u
96.5as
19
79:4:
800
52.544
67.524
6.5g
47.4u
173
4.6
16.40
257ba
FaRRd
75.9
790
89.4o
96-20s
50.0s>
79.5
79.94:
68.1
60.0us
76
214
150
38
40.
1900>
4.6
37.5
25,2os
49.74:
+DCS
76.8
7966
88.9a
96.7es
50.1,
81.0, 700
637pa
7622.
20/7
66
158
53
368
21.3ss
19.2
Table 7: Quantitative evaluations on Llama-2-70B
F Ablation of Sample-level and Class-level Rebalancing
We visualize the ablation of sample-level and class-level rebalancing using TREC (general domain) and PubMedQA (biomedical domain) in Figure and 4b. Both datasets show that sample-level rebalancing significantly elevates weak classes, while class-level rebalancio lancing steadily balances the stronger ones. Ensembling yields overall gains in both metrics.
TREC Accuracy: Initial (ICL) Corrected
PubMedQA Accuracy: Initial (ICL) vs. Corrected
A3
933
1,0
0.8
Accureoy
06
0.6
0.11
+0.24
0.22
0.4
0.32
Ove Tarmshed
0.
Per
Group dess, cample level rebslancing)
diass aumpleitiv neba kncing)
damas des--ccl rebelaroing
desses, class rebalancing)
(a) Accuracy by group of classes, TREC
(b) Accuracy by group of classes, PubMedQA
Figure 4: Ablation of sample-/clas--evel rebalancing
G
Additional Performance Comparisons on Gemma-2-2B
Acc
COBias
Method
Domain
Biomedical
General Damain
Biomedical
All
Avg
AGN
TRP
SST
TRFC
RTF
Avg
DDI
PMQA
All
Aug
AGN
DBP
SST
TREC
RTF
Avg]
TDD
PMQA
1-kket
RCL past-lo delissing
ICL
534
39.0
79.9
72.6uz
27 27.6us
44
71.36s
3944
154gs
63.4a
38.5
3599
25.7as
36bia
42.0ss
512rr
252
44.
35.1
54.43
ONNP
65.1
70.
96.9,
88.4.s
33.25o
69
72.6
52.5
41.5ga
63
19.9
139
7.as
14.h
34.5x
21.4s
300
18,3,
41.8gs
FuRRud
685ps
71.747
553
4800g
67.5gz
12.3
127%
21.6
475
182g
DCS (Ours)
692
745
87.0ss
92.5ts
49:417
70.9 72663
560
46.0s
67.9t
23.3
17.8
77t
68p
30.7cs
21.3a
2,6
37.0
22.42
51.5t.
Table 8: Quantitative evaluations on Gemma-2-25
We additionally experimented with Gemma-2-2B, and obtained consistent improvements (Table 8) similar the Llama cases, showing that our method is applicable LLMs of varied sizes and families.
H More Future Directions
This work deals with prediction biases when directly prompting LLMs. When LLMs or smaller LMs like BERT (Devlin et al., 2019) are finetuned on tasks, which may result in further imbalanced class accuracies, post-hoc corrections can make class accuracies fairer. It would be valuable to understand how these output class probabilities differ from those by direct prompting, and enable custom extensions to the current framework to provide more targeted corrections. It would also be interesting to see how it may help with data-filtered or data-augmented finetuning. In terms of interpretability, those classes that apply class-level correction to any instance is interpretable only at the broader level, so further investigations on how to enforce more interpretability to weight corrections could be conducted.
18


Reviews:
[{"review_id": "y9ifp5npsz", "reviewer": "Reviewer_kpFD", "summary": "The study proposed a post-hoc ensemble debiasing framework to correct in-context learning probabilities for both class and sample levels. It evaluated the performance of the method on 7 datasets, and the results show that the method improves the performance of low-accuracy classes while effectively mitigating class performance imbalances.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method tries to solve the performance of low-accuracy classes while keeping the overall performance at the SOTA level. Results suggest that the method is promising, and it also helps with the domain specific tasks that normally face the challenge of class imbalances.", "reasons_to_reject": "The current study analyzed the annealing time for optimization. It would be helpful if the authors could elaborate more on the effectiveness (e.g., time, cost) of the proposed method for implementation in real practice.", "questions_to_authors": "Please see the previous section for questions and comments.", "extra_scores": {}}, {"review_id": "djnkk3Jb1h", "reviewer": "Reviewer_unSp", "summary": "In this paper, the authors propose to use heaviside step function as the mapping function for  ensemble debiasing to achieve fairer prompting accuracy. Specifically, the mapping function is utilized for in-context learning probability correction. They also propose to solve the non-differentiable framework with simulated annealing algorithm. Experiments conducted on several multi-class text classification benchmark datasets demonstrate that the proposed method can improve the performance of Llama2 LLM. The authors also conducted analysis on probability correct and ablation to validate the effectiveness of the proposed framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The ensemble based correction on weights and membership functions enhance the performance.\n* The proposed method generally improves the performance of Llama2 across many benchmark datasets.\n* Solid analysis on different perspectives in the experiments.", "reasons_to_reject": "* Lack of concrete examples of the proposed method about how it improves the performance.\n* Only test on Llama-2-13B. It would be interesting to see the results on different model families.\n* The proposed framework underperforms in some datasets. It would be interesting to discuss what the gap is.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HisLO0m5Xr", "reviewer": "Reviewer_HyPZ", "summary": "This paper aims to improve the overall accuracy of class predictions by large language models (LLMs) by elevating weaker classes through an ensemble debiasing method based on a Heaviside step function. By formulating class prediction as an optimization problem that accounts for overall error, class-level bias, and sample-level bias, the proposed DCS (Debiasing at Class and Sample levels) framework calibrates LLM predictions to mitigate the aforementioned biases inherited from the training data or caused by prompts, thereby enhancing downstream class prediction accuracy.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. General improved performance across different general domains, especially non-triieveal improvement in the two biomedical domains (Table 1)\n2. It demonstrates the combination of sample-level and class-level correction can elevate the weak classes", "reasons_to_reject": "1. Concerns of over-engineering: this DCS framework involves combinatorial optimization techniques that include many hyper-parameters setup and tuned (e.g. the weight corrections and membership corrections for each class; the simulated annealing (SA) parameters). From the complexity perspective, it may not be that different from directly fine-tuning an LLM on these data, making the performance boost unsurprising. \n2. Generalizability and efficiency: due to the setup and tuning issues mentioned above, DCS is not a plug-and-play framework, it is hard to generalize to a new dataset fast. \n3. Lack of baseline comparison: while this work compare to other optimization-based or fuzzy-based methods, it can be improved by incorporating more baselines. For example, fine-tuning by N-shot example.", "questions_to_authors": "1. The DNIP’s paper (COBias and Debias) and FuRud ’s paper (Let the Fuzzy Rule Speak) are not directly cited when these abbreviation first appear.\n2. The paper can be improved by visualizing the ablation of sample-level rebalance and class-level rebalance by, e.g. bar chart of performance.", "extra_scores": {}}, {"review_id": "8ES1bmjrGs", "reviewer": "Reviewer_vy6v", "summary": "This paper proposes a novel method called DCS (Dynamic Class-wise Scaling) to address class-wise accuracy imbalance in classification tasks. The approach employs simulated annealing to determine optimal linear mapping functions f(·) for probability calibration across different classes. By applying these learned linear transformations to model outputs, the method effectively mitigates class accuracy imbalance while improving overall classification performance. Experimental validation in few-shot settings using LLaMA-2-13B demonstrated that DCS achieved accuracy improvements in six out of seven benchmark datasets, with significant enhancement in the balance of class-wise accuracies.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper demonstrates that the proposed DCS method achieves state-of-the-art (SOTA) performance in addressing class imbalance challenges. In the N-shot setting (where N equals the total number of classes), the approach delivers consistent improvements across six out of seven benchmark datasets, with an average accuracy enhancement of approximately 1%. \n- The paper demonstrates strong writing skills through its well-structured methodology and clear technical exposition.", "reasons_to_reject": "- This paper identifies DNIP and FuRud - two current state-of-the-art methods in the field - at line 249, but fails to formally cite these seminal works in that section. \n- Limited novelty, the study demonstrates limited originality as its core component (the triangular membership function) is directly adopted from previous work by Lin & You (2024b). The primary technical contribution centers on employing simulated annealing to optimize the parameters of these triangular membership functions through a systematic search.", "questions_to_authors": "- Could you clarify how the hyperparameters ak, bk, and ck of the triangular membership function are determined in the paper?", "extra_scores": {}}]

Author rebuttal:

Meta-review: People liked the importance of the problem, the DCS method, and these methods showed very strong results. All reviewers recommended acceptance.
Decision: Accept