Verification and Co-Alignment via Heterogeneous Consistency for Preference-Aligned LLM Annotations
Authors: Cheng Chen
Haiyan Yin
Ivor Tsang
Venue: ICLR 2026 Poster (2026)
Keywords: Verification
Co-Alignment
Preference-Aligned LLM Annotations
Reference-Free Metric
Primary area: foundation or frontier models, including LLMs

Abstract: Large Language Models (LLMs) are increasingly expected to be culturally customisable and personally aligned for natural language understanding (NLU). However, existing methods, from supervised fine-tuning (SFT) to personalised RLHF and prompting, either require costly large-scale annotations or remain constrained by their pretraining distributions. Moreover, acquiring annotations that reflect subjective, diverse, and evolving user preferences is both expensive and labour-intensive. To address these limitations, we propose \textit{\textbf{H}eterogeneous-\textbf{C}onsistency \textbf{C}o-Alignment} (HCC), a training-free annotation paradigm that leverages two heterogeneous models: a knowledge-rich yet potentially overconfident LLM and a task-specialised lightweight model guided by a small user preference set. Together, they verify and co-align misaligned outputs over unlabelled corpora.
For verification, HCC introduces the reference-free \textit{\textbf{C}onsistent}-\textit{\textbf{A}nd}-\textit{\textbf{I}nconsistent} (\textbf{CAI}) Ratio, an uncertainty signal derived from inter-model agreements (consistent samples) and disagreements (inconsistent samples) to determine whether refinement is necessary. For co-alignment, HCC employs a non-parametric, embedding-based preference assignment scheme to recalibrate inconsistent samples according to user preferences.
Across eight NLU datasets and both open- and closed-source LLMs, HCC consistently improves annotation alignment and, in several tasks, enables \textit{Llama-3-8B} to surpass \textit{GPT-3.5/4o-mini} after co-alignment correction. Moreover, CAI strongly correlates with accuracy and tracks pre- and post-alignment gains, offering a reference-free signal for scaling preference-aligned annotation without ground-truth supervision.

Published as a conference paper at ICLR 2026
VERIFICATION AND CO-ALIGNMENT VIA HETERO- GENEOUS CONSISTENCY FOR PREFERENCE-ALIGNED LLM ANNOTATIONS
Cheng Chen Haiyan Yin-.2, Ivor W. Tsang 1.244
1 Centre for Frontier AI Research, Ageney for Science, Technology and Research, Singapore
2 Institute of High Performance Computing, Agency for Science, Technology and Research, Singapore
3 AAustralian Artificial Intelligence Institute, University of Technology Sydney, Australia
*Nanyang Technological University (NTU), Singapore Cheng. chen 160studenn
uts edu au, fyin.haiyan,
edu.
ABSTRACT
Large Language Models (LLMs) are increasingly expected be culturally cus- tomisable and personally aligned for natural language understanding (ÑLU), However, existing methods, from supervised fine-tuning (SFT) to personalised RLHF and prompting, either require costly large-scale annotations or remain constrained by their pretraining distributions. Moreover, acquiring annotations that reflect subjective, diverse, and evolving user preferences is both expensive and labour-intensive. To address these limitations, we propose Heterogeneous- Consistency Co-Alignment (HCC), a training- free annotation paradigm that lever- ages two heterogeneous models: knowledge-rich yet potentially overconfident LLM and task-specialised lightweight model guided by small user preference set. Together, they verify and co-align misaligned outputs over unlabelled corpora. For verification, HCC introduces the reference- free Consistent-A Aaeeeat--nn------niiii Inconsistent (CAI) Ratio, an uncertainty signal derived from inter-model agreements (con- sistent samples) and disagreements (inconsistent samples) to determine whether refinement necessary. For co-alignment, HCC employs a non-parametric, embedding-base based preference assignment scheme to recalibrate inconsistent sam- ples according to user preferences. Across eight NLU datasets and both open- and closed-source LLMs, HCC consistently improves annotation alignment and, in several tasks, enables Llama-3-8B to surpass GPT-3.5/4o-miint after co-alignment correction. Moreover, CAI strongly correlates with accuracy and tracks pre- and vost-alignment gains, offering a reference- free signal for scaling preference- aligned annotation without ground-truth supervision. Code: Ittpp:: thub. Sann//38000333cc/VEEE..
CO- ALIGNMENI AA---AAAL---------EEEEEEEE ICLR26
INTRODUCTION
Demand is burgeoning for culturally and personally aligned LLMs across diverse natural-language understanding (NLU) applications, including culturally aware language understanding (Nguyen et al. 2024), personalized recommendation (Li et al. 2023), household robotics (Han et al. 2024).
and clinical guidance in healthcare (Kadariya et al. 2019 In Southeast Asia, for example, many regional languages feature unique slang, local expressions, and values; stakeholders therefore seek LLMs aligned with local norms and vernacular. In personalized recommendation (Li et al.. (023). users expect models to respect user-specific facts (e.g., names, titles, birthdays, preferences in mu- sic and film) that are not common knowledge. In preference-awar household robotics (Han al., 2024) and clinical guidance (Kadariya et al. 2219)). preferences are proprietary, diverse, and highly individualized (e.g., how dishes are organized, when and where laundry is handled, where cups are stored). Reinforcement Learning from Human Feedback (RLHF) optimizes single global reward that collapses heterogeneous user preferences, which under-represents individual tastes and limiting generalization to unseen, user-specific preferences (Chakraborty et al.) 2024 Ouyang et al. 2022a Supervised finettuning (SFT) can encode specific behaviors when sufficient labeled data are avail- able, but assembling large, high-quality annotations is costly (Ouyang al. 2022b Wei al. 2021 Taori et al. 2023 Tan et al. [2024a] Salemi et al.) 2024 Zhang et al.) 2023a] Tu et al. 2025). To
address personalization with RLHE Poddar al.
2004 propose training latent-conditioned

---

Published as a conference paper at ICLR 2026
Samaaiaa Cf
Podawss
nncrebeun Sarza
Figure I: Schematic overview of Hecerrogeneouus-Conisstteee Co-Alignment (HCC). The inner circle de- notes the consistent set C with higher reliability, while outer region corresponds to inconsistent samples with lower reliability, Each intent cluster is represented by C.. HCC identifies consistency via inter-model com- parison and performs co-alignment using non-parmetrrie, embedding based preference assignment scheme refine inconsistent samples under limited preferenec supervision.
ward model on pluralistic preference corpora and then adapting to new users with a few additional queries. Nonetheless, it still requires training on large, diverse preference corpora. In addition, when a new user's preferences diverge substantially from that training distribution, reward-model generalization may degrade. Yet practice, pretrained LLMs may hallucinate seemingly plausible but incorrect answers due to reliance on population-Ievel statistics (Kalai et al. (2025). Lastly, con- ventional prompt-basee based approaches Wei et al. (2021 2022;; Huang al. (2022) Yao et al. (2022);
Diao et al. (2024); Liu et al. (2023); Wang et al. (203]
[ Yao et al. (2023 Long (2023 Huang et al.
(2023); Madaan et al. (2024): Shinn et al. (2024); Chen Tsang (2024) excel at producing factual
responses to widely known questions, but they remain unreliable for queries involving personal or user-specific information and preferences.
Therefore, it is crucial to build an efficient and economical pipeline for generating preference- aligned annotations from a small, user-labeled preference set and propagating them across large unlabeled corpora which are typically more scalable and cheaper to obtain (van Engelen & Hoos, 2020 Zhu 2020; Chen et al., 2025), so that LLMs move beyond population-level statistics toward user-specife expectations. In addition, conventional reference- based metries (see Table 8 in the Ap- pendix) fall short in reference- free settings, while naive self-evaluation methods within L.Ms often exhibit overconfidence and inconsistencies (Xiong et al. 2023 Stureborg et al. 2024 7hou al.)
2024]. as well as vulnerabilities to prompt format. Moreover, relying solely on the LLM introduces reproducibility issues due to frequent updates of proprictary closed-source models Ito et al. (2025). Collectively, these limitations highlight a critical need for robust, reference- freee evaluation and co- alignment frameworks capable of effectively aligning with nuanced, personalized user preferences in superviid settings. In sum, to train user-preferred, competent LLMs for diverse individual users, especially when only a small number of user-preference exemplars are available, requires two key components: (i) an evaluation framework centered on an interpretable, personalization-awaal reference-free free metric that explicitly accounts for the plurality of human preferences and ensures that generated annotations are accurate and useful. Because collecting large, personalized corpora is pro- hibitively expensive, training evaluators or reward models (e.g., regression models) from scratch is often infeasible (Ito et al.) (025). which underscores the importance of such reference-free metric in the limited-sample regime. (ii) a model-agnostic, low-overhead annotation-aligneel paradigm that effectively rectifies misaligned annotation.
How can we, in general reference-fie setting, assess whether LLM -eneerate outputs are consistent with user pref- erences, without access to ground ground-truth labels or external references?
(i) Reference-Free Uncertainty Evaluation in LLMs:
(ii) Semi-Supervised Co-Alignment for Categorical NLU: How can we design co- alignment mechanism that recalibrates LLM- -eM-ggeeeatedd annotations in categorical NLU tasks using only small set of user preference examples?
To address these challenges, we propose the Heterogeneous Setterogeeoous-Consisennng Co-Alignment (HCC), a training-free free framework consisting of (i) reference- free evaluation and (ii) a co-alignment mecha- nism for
annotations. For evaluation, we introduce the Consistent- ceae--annd-----teeee (CAI) Ratio, a reference-free metric for assessing annotation alignment in preference-based label- ing of unlabeled textual datasets. CAI measures the odds of agreement between two heteroge- neous annotators: (1) an LLM that produces responses driven by next-token probabilities, and (2) mask--pecific/ligtweigge model that operates in embedding space. By cross-checking their out- puts, CAI identifies where token-likehln www.-liiiehhoddeeeeee derived decisions and embedding simiiinitiiilaiittyy judgments agree (consistent samples) versus disagree (inconsistent samples), providing reliability signal that
2

---

Published as a conference paper at ICLR 2026
mitigates the overconfidence observed when relying the LLM alone. For co-alignment, HCC applies MV- VVVV (see f : a nonparametric, embedding- assignment that takes majority votes among the top-k nearest neighbors within clusters seeded by user-preferred exemplars. This propagates preferences to initially unlabeled data. Next, HCC compares the specialized model's as- signed annotations with the L.M''s outputs and performs divide-and-congee andl--nnnd-cerrr co-alignment (DCCA), again using MV-VTES, to repair disagreements. Crucially, HCC avoids dependence on ad-hoc con- fidence thresholds derived from next-token probabilities, thereby reducing overconfidence errors. Our main contributions are as follows:
We propose Heterogeneous-Consicee Consistency Co-Alignment (HCC), a novel training- free framework that (i) enables reference-fiee free uncertainty evaluation in LLMs and (ii) provides an efficient co-alignment mechanism to recalibrate LLM-produced annotations in categor- ical NLU tasks using only small set of user preference examples.
We introduce the reference-free quantifies inter-model agreement as a proxy for weeference-aligmment uncertainty, enabling evaluation without ground-trui truth labels and addressing the challenge of assessing LLM- generated annotations under limited user-labelled examples.
(CAI) Ratio, a metric that
We validate the effectiveness of HCC eight domain sppcciiii categorical NLU datasets, demonstrating substantial and consistent improvements in annotation alignment.
2 RELATED WORKS
LLMs for Data Annotation. Large Language Models (LLMs) have demonstrated strong capabili- ties in text and data annotation across a wide range of open-domain tasks (Meng et al. 2022 Ye et al.)
2022 Wang et al. 2024 Liu et al. 2024; Wu et al. 2024), including spoken language understanding (Chen et al.) 2023 2024). ). In many cases, LLM-generated labels rival or exceed crowdsourced and
manual annotations without task-specific training (Gilardi et al. 2023). Consequently, such annota- tions have been widely adopted for supervised fine-tuning, alignment optimisation, and downstream inference (Tan et al., 2024b). However, relying solely on LLMs for annotation can be risky. With- explicit verification mechanisms, LLM outputs may exhibit overconfidence, systematic bias, or misalignment with user-defined preferences (Xiong et al., 2023 Stureborg et al. 2024] Zhou et al.) 2024]. This concern becomes particularly salient in preference-aligue
NLU settings, where annota- tions must reflect personalised or task-specific intent. In practice, many real-world scenarios operate in semi-supervised regime, where a small set of labelled preference examples guides annotation over large unlabelled corpora. While this setting reduces the need for extensive supervision, reli- able evaluation remains challenging when labelled data are searce. Moreover, when the unlabelled corpus is large, repeatedly querying closed-source LLMs may be prohibitively expensive, whereas smaller open-source models may lack sufficient reliability for direct deployment. These challenges highlight the need for training-free free frameworks that combine strong or weak LLMs with accessible sss-ispecillieed models, while incorporating reference-fir free evaluation and alignment mechanisms.
Using Specialised Models to Guide LLMs. While pretrained task-specific models can efficiently annotate unlabelled data using few examples, their performance often degrades when user prefer- ences diverge from the original training distribution. Consequently, using these specialised models to guide LLMs is risky: it implicitly assumes the model's predictions are preference-consistent. Without verification, these errors can propagate to the LLM, amplifying misalignment. A similar flaw exists in conventional clustering methods, which assume geometric proximity in embedding space ensures semantic correctness. In practice, personalised preferences rarely align perfectly with embedding homogeneity. Even LLM-guided clustering methods like ClusterLLM (Zhang et al, 2023b) remain bound by initial embedding hierarchy, ultimately lacking a principled, reference- free mechanism to verify and correct preference misalignment under limited supervision. Con- versely, HCC treats embedding similarity as structural prior for propagation rather than proxy for correctness, and it retains or revises annotations based on the agreement between the LLM and embedding-induee induced hypotheses, thereby avoiding blind reliance on geometrie proximity.
Prompt-based Approaches. Praeet methods leverage in-context learning to guide LLM outputs without parameter updates. Self- Colf-Consistency (Wang al., 2022) enhances reliability by sampling multiple reasoning paths and selecting the most consistent output. While effective in im- proving response accuracy, this approach assumes the underlying L.M is sufficiently capable and
3

---

Published as a conference paper at ICLR 2026
typically requires repeated sampling, resulting in substantial computational overhead. Moreover, consistency across reasoning paths does not guarantee alignment with user-defined preferences. Shain-of-Thoughhe (CoT) prompting (Wei al. (022) improves interpretability and performance by explicitly modelling intermediate reasoning steps, while Few-shot Prompting (Brown et al., (020)) incorporates small number of illustrative examples to steer outputs toward task-specific intent. De- spite their empirical success, these methods primarily rely on the intrinsic capabilities of the LLM and the quality of prompt construction. They lack explicit mechanisms for reference-free reliability assessment or systematic co-ar aliigmennt with personalised preference distributions under limited su- pervision. KATE (Liu et al., 2022) improves prompting by retrieving the top-k semantically similar labelled examples as demonstrations. However, when the seed labelled set is small, the retrieved examples may fail to capture the diversity and ambiguity of user preferences, limiting its adaptabil- ity to personalised annotation tasks. Finally, Self-Refine (Madaan et al. 2024) iteratively improves initial outputs through self-correction. While promising. its success depends heavily on both the capability of the LLM and the suitability of any external tools used for the target task.
3 PROBLEM SETTING
Let the unlabeled training corpus and the test corpus be denoted D, = {21, ......... ZN and Dest
".q....,T',. where each instance satisfies x E X' S Rd A small set of user peer-prfference annotations (see Section 4.1 serves as the alignment reference set, with the goal of propagating a preference label ÿ E y 1,.... Y to each x E Du-- We assume that the distribution D., can be partitioned into two subsets: consistent samples C and inconsistent samples I, such that C, I Du, CnI W. and [C] + [Z] = [Daul In practice, the subsets are unknown and must be estimated (see Section $4 using specialized embedding-basse
model S and an LLM T. given a small user-preference set H. Formally, H is clustered into & disjoint subsets (CI...,,Ca Ck each representing semantically coherent region of the embedding space. Alongside the samples, set of preference annotations is also given, denoted as y {.1. 1/, ) Hence each cluster is formally denoted as Cj {(r., Vj) Iz E H where H, S H and H ((x: ū)) with = 5% of [Du)l The clusters are disjoint, satisfying nC, = 0, vi # j). and their union fully covers H. Our goals are: (i) to assess LLM annotation alignment by identifying latent consistent and inconsistent subsets C* and It and (ii) to improve annotation accuracy and alignment according to user preferences.
4 CO-ALIGNING LLMS WITH Heterogeneous- Consistency Co-Alignment
HCC leverages agreements (consistent samples) and disagreements (inconsistent samples) between two heterogeneous models: an LLM annotator, which generates responses from token-level prob- abilities, and task-specific model, which captures fine-grained embedding similarities. Cross- checking both models identifies samples where token probabilities and embeddings align, mitigating the overconfidence of LLM-only predictions. For co-alignment, HCC employs MV-VTES, a non- parametric, embedding- based preference assignment that uses majority voting the top-k near- est neighbors within clusters initialized by user- uref-rree samples, enabling annotation of unlabeled data. Finally, HCC refines inconsistent samples via a sivide-and-conquer co-alignment (DCCA) strategy, again combined with MV-VTES.
4.1 SEMANTIC CLUSTERING BASED ANNOTATION FOR UNLABELLED DATA
We propose a semantic clustering approach propagate and refine annotations over unlabelled data. By applying majority voting over top-k nearest neighbours in embedding space (see Section 4.3.2). labels from small prefereece reference set are extended to semantically similar instances. This structured propagation provides local contextual signals for detecting and correcting misaligned annotations, thereby supporting co-alignment in semi-supervised settings.
4.1.1 TASK-SPECIFIC SPECIALISED MODEL
We adopt MiniLM (Wagg et al. 2020), sentence-tranee transorerr model, as the task-specialised model, denoted by S. The model encodes each instance x; into a sentence embedding: S(z,) = e;. We em- ploy semantic clustering-based assignment scheme, where annotations assigned according to
4

---

Published as a conference paper at ICLR 2026
similarity in the embedding space without additional model training. Each cluster Cj consists of em- beddings of samples associated with label j. To measure alignment between an instance and cluster Cj. compute the average cosine similarity over the top-k most similar embeddings in that cluster:
AS(e, Cj) =
Σ
Teilllee|
e
(1)
Uniabaied Tisiring Corpus
Uitse dir Preferenci
Steeet Stet
Ansiga Chastex Cy
((C,,))(C
Here, e, denotes the embedding of 24, and represents embeddings of samples in cluster Cj. The operator Top-k(C,, selects the samples in Cj with the highest cosine similar- ity to e,, In our experiments, we set to five. The cluster assigned to 2; is determined by
EG
C, arg AS(a,,
Encoder (MiniLM) Model:
8(z))
8(z,)
Uollel
Cj- arg max AS( Cÿ)-
(2)
Emreeded
E "argaiiee
The corresponding cluster label Ty- is then as- signed 24. i.c., 3: yy.. This defines the annotation function S(x;) induced by the spe- cialised model. The resulting annotated dataset is Ds = {(z,, ) where y
Aliezmonn Souree A8(
Figure 2: Semantic Clustering -Based Annotation Assignment Scheme Based on Embedding Simi- larity in Task-Specialised Model.
(S) denotes the annotation assigned through this semantic clustering-basee based assignment scheme.
4.1.2 GENERAL LLM ANNOTATION
Given the specialised-ment model annotated dataset Ds. we query a large language model T to generate annotations via group prompting. We consider both zero-shot and single-shot settings.
Zero-shot. The LLM generates annotations independently for each input: o! T(x:), with
autoregressive likelihood P( (T) [z))) 11 P
(T)
yi
[4, Di,es) where L, denotes the length of
the generated sequence and j represents previously generated tokens. This produces the LLM- (1) >
annotated dataset Dr {2, Di Single-shot. We incorporate the pnecialised-mode annotation contextual guidance: o! vl (9)
T( (5)), with likelihood P(D!" (2) (1,,)())
II P Ui (na,
(..2)) This yields the
augmented annotated dataset Dr = {1I,, 11 )
4.2 IDENTIFICATION OF CONSISTENT AND INCONSISTENT SAMPLES
After obtaining the specialised-mode dataset Ds. the LLM-generated dataset Dr. and the augmented dataset Dr, we introduce paasistent-and-Innoniin agreement-based procedure that partitions samples into consistent and inconsistent subsets via cross- model annotation comparison. Based on this partition, we define the CAI Ratio as a reference- free reliability metric that quantifies annotation stability by measuring the relative proportion of consis- tent samples within the unlabeled corpus. In contrast to conventional clustering similarity measures, such as the split and merge framework of Xiang al. (2012), which decomposes partition similarity via weightee local aggregation, CAI applies an instance-level aggregation principle to quantify inter-annotator consistency. For each æ E Du,, the annotation label from the task-specialised model denoted as g and that for the LLM is denoted as g(T)) (zero-shot) and ÿ(T)) (single-shot). A sample is considered consistent if:
(CAI) Identification, an
CC, where C represents the set of consistent samples, while conversely, sample is considered inconsis- tent if at least one of the annotations differs:
g($) (=) =()))
3 (v.v') V E (5) (T)) o (T),, a v 3 ZEI,
5

---

Published as a conference paper at ICLR 2026
Incondicant Sample (The Annotaatio on g2, (Pttmotton)
Samole
ILLMs Anmotattin geeerstionn
(tru Areotaaion gocardiion
Sarrple
Condibtent Sample JuMM Armotiation on go s.srrotion)
Datasats
Oreade
occombl
Figure 3: An illustrative figure highlighting the sample
Figure 4: Visualization of t-SNE Clustering (better viewed color, enlarged) comparing LLM vs Ground- Truth Annotations on Go Emotion Dataset. L.M out- puts exhibit high similarity with ground-truth labels on consistent samples, while showing significant diver- gence on inconsistent samples.
importance of consistent-and-inconiiis identification in evaluating LLM performance. LLM annotations on inconsistent samples (dark- colored bars) exhibit significantly lower accuracy compared to those on consistent samples (light- colored bars).
CAI Score LLM Accuracy (GPT-3.6) 6)
CAI Score vs LLM Accuracy (CPT-40 Min)
CAI Bcore LLM Accuracy (Google Gamin)
CAI Score LLM Accuracy (Liama, 3B)
CAI Scor
CN
Score
CA Soore
CAU Boors
Table I: Correlation analysis between LLM annotation accuracy and the CAT ratio, evaluated across four principled LLMs (also see statistical test results in See 6.]]. The Pearson correlation coefficients and corresponding p-values confirm the statistical significance of the positive correlation between the CAI ratio and LLMs accuracy.
where I represents the set of inconsistent samples. Identifying annotation inconsistencies and eval- uating the reliability of LLM outputs without ground truth remains central challenge in semi- supervised LLM annotation. Our HCC framework addresses this by leveraging disagreement be- tween the task-specialised model and the LLM to identify unreliable annotations and trigger alignment. We introduce the Consistent- Inconsistent (CAI) Ratio, a novel HCC metric for as- sessing the trustworthiness of LLM- --geeeertteed labels without access to ground truth. For the formal definition of CAI, we fix the LLM hypothesis (either zero-shot or single-shot), ensuring consistency with Definition 4.1 and Proposition 4.2.
Definition 4.1 (Consistent-and-Inet Innosssssntnnnnnnsteette (CAD) Ratio) Let D., { be an unlabelled dataset. Given an LLM hypothesis T and a cask-specialised model hypothesis S. define C= {x:; E Du T(xa)= = S(x:)), I {zx E D. T(x;)) + S(x.)). Let No == IC| and NIc := II]. No
The CAI Ratio is defined as CAI(Da; S) = Nio+s. where & > is a small constant to avoid division by zero.
The CAI Ratio leverages heterogeneous consistency, combining likelihood-derived signals from the LLM with ambedding-based agreement from task-specialised model, to evaluate the reliability of LLM-generated annotations without labelled supervision. A markedly high CAI Ratio (CAI > 1) may indicate systematic agreement that could stem from overconfidence a low CAI Ratio (CAI < 1) reflects frequent disagreement and thus elevated uncertainty in the model's outputs. In both regimes, the ratio serves as a reference-free indicator of annotation align-
shared bias, whereas

---

Published as a conference paper at ICLR 2026
ment, guiding decisions whether LLM-generated labels should be refined through additional hu- man supervision or enriched prior knowledge. Furthermore, we formalise the relationship between the CAI Ratio and annotation accuracy through the Consistency Principle. Under mild assumptions that the L.M and the task-specialised model share a common latent preference-aligned target and exhibit bounded disagreement, the proportion of consistent samples is expected exceed that of inconsistent samples as the dataset size [Da grows. This asymptotic behaviour provides theoretical support for interpreting CAI as a proxy for annotation reliability.
Proposition 4.2 (Consistency Principle) Let Du, {ma)== be i.i.d. samples drawn from D. Let T and S denote the LLM-induced and embedding-induue iinduiid hypotheses, respectively, and assume both are optimal under the same target distribution. Define Pagree == Prz=D (x)) = =S(x)] and pdis = PraND] (x) # L(z)),Let No = >i=1 (11111((x))) = S(x,)) and NIc = Ei=1 [Tx)), # S(x,)), If
Pagree > Pdis, then Du- Pr(Nc Nic)= 1.
4.3 HETEROGENEOUS CONSISSTENY CO-ALIGNMENT
Given the consistent samples with high-quality annotations C [(x, 7 and the inconsistent samples I = ((r; ot )) identified via CAI, together with the user-preference set H, we perform co-alignment of the annotations in I using a divide-and-conquer strategy. Here, of and of denote the annotations associated with consistent and inconsistent samples, respectively. As illustrated in Fig. 3] the samples in exhibit substantially higher accuracy than those I. We therefore leverage the reliable annotations in C to guide the correction and realignment of the inconsistent samples in I.
4.3.1 DIVIDE-AND-CONQUE
CO-ALIGNMENT (DCCA)
Round 1. (co-aligning I to obtain I())). For each (x. ÿ²) E I,, we apply the (MV-VTES) (Sec- tion 4.3.2) using the reference set C U H. Let ÿ denote the reassigned annotation produced by MV-VTES. We define [ {2x, 0) (x, consistent samples after first-round reassignment. We then perform CAT identification by compar- ing the original annotation g' in I with the updated annotation ý I(1,, yielding the partition I == CI U II, II,CIr n II 2. Here, CI contains the samples that become consistent after the first alignment pass, while II contains those that remain inconsistent.
Σ). Thus, I(1) represents the originally in-
(Co-aligning II to obtain II For each remaining (x. E II, we again apply MV-VTES, this time using the expanded reference set CUCIU U H. Let ÿ denote the newly assigned annotation, and define (1) {(z, 5) (x. 5 E II). After the second pass, the fully aligned version of the originally inconsistent set ( = CI U (I(1), Finally, the self-corrected dataset is (fiina) = U CI U (1) Overall, DCCA partitions into CI and II, resolves the easier inconsistencies in the first pass and the more challenging ones the second, and produces the refined dataset D (final) through iterative co-alignment.
Round 2.
4.3.2 MAJORITY VOTING VIA TOP-NEAREST EMBEDDING SCHEME (MV-VTES)
Our divide-and-conquer co-alignment (DCCA) strategy relies on the MV-VTES method, which re- fines annotations of inconsistent samples by selecting labels from the most semantically similar references in C U H. Given a query sample æ Du. we first determine its consistency status using CAL. For sample identified as inconsistent (xr 1), we refine its annotation via MV-VTES using the labeled reference set D_ = C H C X x Y, where each element a pair (a, 7) consisting of input and its high-quality annotation ÿ. Let A fa X' (a, Ø) D.} denote the set of reference inputs. We retrieve the top-k most similar reference inputs to x according to cosine similarity in embedding space:
S(a) S(x) (s())) eeA. [S(a) [
{ = TopK
where S(-) denotes the embedding function. For each retrieved imput a,, let ÿ denote its associated annotation D... The refined annotation ÿ for x is then obtained via majority voting over the retrieved labels. Specifically, let A (y1,-.., Wkk be the set of unique labels among the retrieved
7

---

Published as a conference paper at ICLR 2026
neighbors. For each E A, define its frequency na Σ Il a). assigned as
The refined annotation is
i arg max a. a
(4)
Hence, our method treats consistent samples structural anchors for correcting inconsistent ones.
Rather than discarding noisy or conflicting annotations, HCC systematically co-aligns them to im- prove global annotation consistency.
EXPERIMENTS
We conduct extensive comparisons against the following baselines: (1) Specialised Model (Reimers)
& Gurevych, 2019
(2) LLM Grattafiori et al. (2024); (3) Specialised Model+LLM: (4) Clus-
tering Zhang et al. (2023)): (5) CoT Wei et al. (2022); (6) FoT [Brown et al.] (2020); (7) Self-
Consistency Wang et al. (2022); (8) Self-Refine Madaan et al. (024)). Details on our baselines,
datasets, and evaluation metries shown in Appendix Daad
(a) GPT-3.5 Turbo (Closed-source LLM)): HCC achieves the best accuracy on 5 datasets.
Dutesct
Sper. Medef"
LLM Zemoshet 31
Spe.alLM
Chat!!
Cup
Fup
SellCren
S
BCC Ciee
BCC Ciee CAldBelore Aferr
Mazoee Scenards
75 13.00125
18 18
9188aca:
18.4...
47.0lss
60.35..
150 4422
MTOP hatee
64 11
0400 x
Banling77
73.00
65 68.12
13.00axs
06.00..m
82.
120 -015
Rodde
22
05.30..
58.7a
0.00 -1.0
Musire bntew
53 18.00
71.02.00
64.44u0
0155usm:
02.221m
74.8...
12.4...11
L12 -241
(b) GPT-40 Mini (Closed-souree LLM)): HCC achieves the best accuracy on 6 datasets.
Spen
ELMiZenshh
Spe
Sef.Re
BCC
(Belor-AAtte)
Clinr
51 Hasus
29 79.38u
71
17 1ox
01. a0.r
06.00
AT Ml
2.005.30
Stmmanii
25-55 m6
06 Mn
77 Ma
0180
06 Raam
79 m0
An 18
1-30-10
MTOP Invnt
32:55 53
15 M..0t
25.5
Rodking?
7130u8
05 17
71 71300
5633
m 0.
42 45
1.32- (13
Redile
57 00
383
1::0-1:0
Nat
Mossiry Anient
01.00.111
7630
0100aa1 66.4
76.0
58.220012
78
72.00ua
1.47-2.33
(c) Meta-Llama 3-8B Instruct (Open-source LLM): HCC achieves the best accuracy on
datasets.
Spec.
[Hefore-
Cline
32.
51 51.07aa
52.8.a
18 18.02..oo
N2 82-48
[150-1 (3
Maroiee Scenario
75 T5.52a1
43 43.52:1.0
56. Tezon
43.29es
41.30g133
58.11.na
54
70.0
75 18
0.07 1077-4..0 58
MTOPP Inzeut
12 2222222
34. 34.17amm
46 20u20
di.3Paea:
08.18.42
30.Gacc
08.39
55 55.20
[150-1.0 0n
Etchane
20 27
13.00um
15 ss
16.04
5.
16.00gn
0.51-0
Banking?
13.00.100
24
32.88ue.
58.00
30.0
Reader
Fewket Nat
14.20.004
50.0
Marohe Intent
46,41are
56.0
55.02gm
41
74.05.0.1
$4.0
67.40
57. 57.70gm
0.78 12.87
Table 2: Comparative Performance Across LLMs Accuracy (%) on eight benchmark datasets for
GPT-3.5 Turbo, GPT-4o Mini, and Meta-Llama 3-8B Instruct. We compare (i) zero-shot inference, (ii) prompting and clustering baselines, and (iii) HCC before and after correction. HCC achieves the highest or near-highest accuracy on most datasets (5-7 per model) and consistently increases the CAI score (Before After) indicating stronger cross-model consistency. Results are reported as mean + standard deviation over three runs.
5.1 EVALUATION RESULTS
GPT-3.5 Turbo: We have two key findings based on the experimental results from Table 2 First, our method (HCC) consistently outperforms baseline methods, achieving notable gains on Clinic (..17%), Massive Scenario (+0.88%), and Bank77 ++2.99%). ). For the MTOP Intent and StackExchange HCC surpasses the non-collaborative baselines. i.c., Only Specialised Model and Only LLMS (GPT 3.5). by +16%/+4.111% % and +9.18 9 35%, respectively. This underscores the effectiveness of HCC in improving annotation accuracy through the collaborative refinement of a specialized model and an LLM. In particular, knowledge distillation between the spe- cialized model (BERT) and the LLM using consistent samples (denoted as HCC w/o Corr) achieved the highest annotation accuracy on the MTOP Intent dataset. GPT-4o Mini: Based on the results b. b, there are two key findings. First, Heterogeneous Satroggennouu-CCnssttennny Co-Aligmment (HCC)
from Table consistently outperforms all basclines on cline (+2.7%). Massive Scenario (+0.58 % ) Bank77 (+7.06%% % and Reddit (+3.92%). Additionally, HCC without alignment achieved the
8

---

Published as a conference paper at ICLR 2026
Dataset
HCC (MiniLM)
HCC (BERT)
GPTT-4o Mini
GPT-3355 Turbo
MiniL. Outcome
BERT Outcome
CLINC
82.43
82.40
81.44
66.58
Beats Both
Beats Both
Massive Scenario
78.13
74.61
66.83
60.89
Beats Both
Beats Both
MTop Intent
63,39
65.62
75.03
64,95
Lower than Both
Beats 3PT Only
StackExchange
38, 88
39.41
51,90
30.10
Beats GPT:3.5 Only
Reats GPT-3.5 Only
Banking77
77.71
71.62
65,12
65.12
Beats Both
Beals Both
Reddit
58.81
58.13
$7.40
51.12
Beats Both
Beats Both
FowRel-Nar
42.92
42,05
35.87
32.87
Beats Both
Beats Both
Massive Intent
67.75
62.61
76,93
71.52
Lower than Both
Lower than Both
Table 3: HCC under MiniLM and BERT backbones compared with closed-source LLMs.
highest annotation accuracy on MTOP Intent. HCC (w/ Corr) and HCC (w/o Corr) can be used interchangeably to enhance annotation accuracy.
Meta-Llamaa-833 Instruct: HCC with correction achieves the highest accuracy on 7 of the 8 bench- mark datasets. Despite the relatively weak zero-shot performance of Llama3-8B, our method sub- stantially improves both accuracy and CAL, often surpassing the base LLM and the standalone spe- cialised model. These results indicate that HCC remains effective even when the underlying LLM is comparatively weak, showing strong robustness to model quality.
5.2 HCC UNDER WEAKER (MINILM AND BERT- BASE-UNCARA AND STRONGER (LLAMA-3-8B INSTRUCT) MODEL REGIMES
Overall, HCC consistently outperforms prompt-based baselines such as Self-Refine. Prompting methods often underperform categorical NLU tasks: unlike reasoning benchmarks where ecutable code can verify solutions, NLU lacks an external correctness oracle, and self-generated feedback may
degrade annotation quality (Huang et al. 2023). Moreover, prompt-based ap- proaches depend heavily on strong proprietary [..Ms, with performance degrading on more com- plex tasks or weaker open-source models (e.g.. Llama-3-8B). To evaluate robustness across back- bone capacities, we pair two task-specialised models of different wrengths-Minin (stronger) and BERT-Base-Uncase
(weaker), with L1 ama- 3-8B Instruct. This setting tests whether HCC remains effective under varying structural representation quality.
BERT-Base-Uncased As shown in Table 3 HCC significantly improves annotation quality even with this weaker backbone. In this regime, I.1 outperforms GPT-4o Mini on five datasets and surpasses GPT-3.5 Turbo on seven datasets. These results indicate that HCC remains effective even when the specialised encoder provides com- paratively weaker structural signals.
Inst ruct combined with HCC
MiniLM. When paired with the stronger MiniLM backbone, HCC further improves performance consistency. Under this configuration, HCC combined with amaa 1amm---5-0 3--8B Instruct surpasses GPT-3.5 Turbo six datasets and outperforms GPT-4o Mini i on five datasets. This suggests that stronger structural representations amplify the gains from heterogeneous consistency while pre- serving robustness across datasets. Overall, these findings demonstrate that HCC enables competi- tive performance even when paired with weaker open-source L.MSs, reducing reliance on proprietary models.
5.2.1 IMBALANCED USER PREFFREREE SAMPLES
In real-world scenarios, the distribution of user-prefereed samples is often imbalanced. To assess the robustness of our method, we conduct comprehensive evaluation of HCC under various annotation budgets, 1 %%, 5%, and 10% in the presence of imbalanced label distributions. We perform abla- tion studies to examine the impact of user preference imbalance, as shown in Table (Appendix), using Llama 3-8B Instruct under an imbalance ratio of 60%. Specifically, after assigning one beled sample per class, 60% of the remaining samples are allocated to the majority classes. This setup reflects a moderately skewed yet realistic user preference distribution. HCC assumes at least one labeled instance per class to align annotations with user intent. Despite the imbalance, HCC consistently outperforms strong baselines such as Self-Consistency (+29.58% on Clinc, +33.25% Self-Refine (+34.09% Clinc, +37.63%
on St Stackexchange. +22.56% on Pew_Rel_Nat on 77)) and Clustering (+10.03% on Banking77, +22.26% on Reddit), all under bal- anced 5% label supervision. However, HCC does experience some performance degradation due

---

Published as a conference paper at ICLR 2026
to the imbalanced setup, with accuracy drops observed Clinc (-0.32), Massive-Scenard (-0.46), Stackexchange (-0.59), Reddit (-0.11), 11), and Few_Rel Nat (-0.52). Overall, the HCC framework demonstrates strong robustness under moderately imbalanced conditions.
CORRELATION RESULTS BETWEEN CAI RATIO AND LLM ACCURACY
We performed Pearson correlation analysis to investigate the relationship between the CAI Ratio and LLM accuracy. The correlation analysis between the Consistent-and-Iie and-nncossisten (CAT) ratio and accuracy across three different LLMs demonstrates strong relationship between these two metrics. GPT-3.5 shows the highest correlation (p = 0.93. p = 8.22 x 10-5), indicating a very strong positive relationship between CAI and accuracy, with high statistical significance. GPT-4o Mini shows a strong correlation (p = 0.86, 1.61 x 10 10-3), suggesting that CAI is a reliable indicator of LLM accuracy for this model. Llama-8B-Insten
(p 0.81, p 1.44 x 10 3) and Google Gemini (p = 0.72, 1.80 x 10 10) exhibits moderate-to-strorm correlations with significant statistical confidence.
CAI RATIO EVALUATION AND ANALYSIS
As shown in Table 2 applying HCC consistently increases the ratio across GPT-3.5 Turbo, GPT- 4o Mini, and Meta-I Jama3-8B Instruct. In many cases, substantial CAI growth coincides with no- table accuracy improvements. For closed-source models (GPT-3.5 and GPT-4o Mini), large CAI in- creases are typically accompanied by strong performance gains. Under GPT-4o Mini, Banking77 improves from CAI 1.36 4.03 with accuracy rising from 65.12% 82.45%. Similarly, Clinc increases from 2.06 to 5.20 (81.44% + 87.93%), and Mass Massive Scenario from 1.39 to 4,65 (66.83% + 80.18% 3). A similar trend observed under GPT-3.5 Turbo, where CI Clinc, Massive Scenario, and Banking77 exhibit both strong CAI growth and corresponding accuracy gains. For the open-source Llama3-8B model, the effect particularly pronounced. Although zero-shot Llama performance is relatively weak, HCC substantially increases CAI (e.g., 0.56 + 4.43 on Clinc, 0.67 - 4.88 on Massive Scenario). with large corresponding accuracy improvements (32.49% + 82.43% and 43.52% + 78.13%. respectively). This demonstrates that HCC remains effective even when the base LLM exhibits limited initial consistency. However, the relationship is not strictly monotonic. On MTOP Intent under GPT-4o Mini, CAI increases from 0.74 to 1.66, yet accuracy decreases from 75.03% to 67.10%. Similarly, on StackExchange, CAI improves modestly (e.g.. 0.31 0.66), while HCC remains below the LLM-only baseline. These cases indi- cate that CAI growth alone does not guarantee accuracy improvement. Empirically, substantial CAI increases, particularly when CAI moves well above are often accompanied by meaningful per- formance gains, whereas marginal increases tend to provide limited structural reinforcement. HCC generally most effective when the initial CAT already reflects moderate cross-model agreement (e.g.. CAT >> for closed-souree models or > 0.5 for weaker LLMs), suggesting that co-alignment amplifies existing structural consistency rather than creating it from scratch. Across model families, HCC achieves the highest accuracy on the majority of datasets, showing consistent effectiveness across both closed- and open-source LLMs.
7 CONCLUSION
This paper addresses reference- free evaluation and co-alignment in semi-supervised NLU annotation under limited user preference supervision. We propose Heterogeneous Consistency Co-Alignment (HCC), a training-free framework that leverages agreement and disagreement between an LLM and task-specialised model assess and refine annotation quality. HCC introduces the CAI Ratio as a reference- indicator of alignment and employs divide-and-congee recalibrate inconsistent samples. Extensive experiments across eight NLU benchmarks demonstrate that HCC consistently improves annotation alignment. Notably, HCC enables competitive perfor- mance for open- LLMs when compared with proprietary models. Furthermore, the CAI Ratio correlates strongly with observed alignment gains, supporting its role as a scalable reference frrence--reeeeee evaluation signal. Future work will investigate theoretical extensions of heterogeneous consistency under broader preference modelling settings.
co-alignment strategy to
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work studies preference-aligned annotation through heterogeneous verification and co- alignment, without additional model training. All experiments are conducted on publicly available datasets and do not involve human subjects, private data, or high-stakes deployment settings.
REPRODUCIBILITY STATEMENT
We provide comprehensive documentation ensure full reproducibility of results. The com- plete Heterogeneous Consissennyy Co-Aligmment (HCC) procedure is specified in Appendix R (Al- gorithm Table), detailing the CAl-based identification process, verification mechanism, and divide- and-conquer co-alignment refinement steps. All prompting templates used for LLM annotation are reported in Appendix (Prompt Instructions), including task formulations and preference- seed inte- gration strategies for both zero-shot and single-shot settings. Additional implementation details, in- cluding experimental configurations and hyperparameters, are provided in Appendix G. All datasets used in our experiments are publicly available (Appendix D), and all evaluation metrics, includ- ing the proposed CAI Ratio, are formally defined in the main text and Appendix F. No proprictary datasets, hidden annotations, undisclosed training procedures are involved.
ACKNOWLEDGEMENTS
This work was supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP-2024-00 findings and conclusions or recommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Any opinions,
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. arXiv preprint 0rXiv:2003.04807, 2020.
Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Dinesh Manocha, Furong Huang, Am- rit Singh Bedi, and Mengdi Wang. Maxmin-rlhf: Alignment with diverse human preferences. arXiv preprint arXi::2402.0892 2024. CRL.httts://aart
wwtp:/gg////bo.o...ooo..oo..ssss abs /2022.22222222 08925)
Cheng Chen and Ivor Tsang. Self-teaching prompting for multi-intent learning with limited super- vision. In The Second Tiny Papers Track at ICLR 2024, 2024. URL /RLLitttp:://oeeevviie
net/forund wod.:oomdmddedammmmm
Cheng Chen, Bowen Xing, and Ivor W Tsang. Low-hanging fruit: Knowledge distillation from noisy teachers for open domain spoken language understanding. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pp. 125, 2024.
Cheng Chen, Atsushi Nitanda, and Ivor Tsang. Unlearning misalignment for personalized LLM adaptation via instance-repooseeddeeeed discrepancies. Transactions on Machine Learn- ing Research, 2025. ISSN 2835-8856. URL https: patp::://opentee http////ppeew.eettt.te forum?id= n BJE3sWPBMc
Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. arXiv preprint arXiv:2330..55128, 2023.
Shizhe Diao, Pengcheng Wang, Yong Lin, Rui Pan, Xiang Liu, and Tong Zhang. Active prompting with chain-of- thhuuught for large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1330-1350, 2024.
11

---

Published as a conference paper at ICLR 2026
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natarajan. Massive: A Im-example mul- tilingual natural language understanding dataset with 51 typologically-divrse languages, 2022.
Gregor Geigle, Nils Reimers, Andreas Rücklé, and Iryna Gurevych. Tweac: transformer with ex- tendable qa agent classifiers, arXiv preprint arXiv:2104.0708) 070001 2021.
Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. Chatgpt outperforms crowd workers for text- annotation tasks. Proceedings of the National Academy of Sciences, 120(30):230501612) 2023.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21755 2024.
Dongge Han, Trevor McInroe, Adam Jelley, Stefano V. Albrecht, Peter Bell, and Amos Storkey. Lim-personalize: Aligning IIm planners with human preferences via reinforced self-training for housekeeping robots, 2024. URL htt/htts::///at //appiss//rxiv...///// org/ httgg/bb//244444 14285
Xu Han, Hao Zhu, Pengfei Yu, Ziyun Wang, Yuan Yao, Zhiyuan Liu, and Maosong Sun. Fewrel: A large-scale supervised few-shot relation classification dataset with state-of-the of-tthearrt evaluation. arXiv preprint arXiv: 1810. 10147, 2018.
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang. Hongkun Yu, and Jiawei Han. Large language models can self-improve. arXiv preprint arXiv:2210. 11610, 2022.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2023.
Takumi Ito, Kees van Deemter, and Jun Suzuki. Reference free evaluation metrics for text genera- tion: A survey. arXiv preprini arXiv:2501.1201 2025.
Dipesh Kadariya, Revathy Venkataramanan, Hong Yung Yip, Maninder Kalra, Krishnaprasad Thirunarayanan, and Amit Sheth. kbot: knowledge-enable personalized chatbot for asthma mafa-mannaeeent. In 2019 IEEE International Conference on Smart Computing (SMARTCOMP), pp. 138-143. IEEE, 2019.
Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, and Edwin Zhang. Why language models hallucinate, 2025. URL haltaat///arxiy orw.//oorrrooooo abs/ 04664
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Kentaro Inui, Jing Jiang. Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empir- ical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 1311-1316, Hong Kong, China, Novem- ber 2019a. Association for Computational Linguistics. doi: 10. 10.188653/1/DD19-1111 URI. https::/accaatthir htt.:aaattt.ooooooooooD//-------
Stefan Larson, Anish Mahendran, Joseph J Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K Kummerfeld. Kevin Leach, Michael A Laurenzano, Lingjia Tang, al. An evaluation dataset for intent classification and out-of-scope prediction. arXiv preprint arXiv: 1909902002, 2019b.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. Mtop: A comprehensive multilingual task-oriented semantic parsing benchmark. arXiv preprint arXiv:2008. 09335, 2020.
Jaachnng Li, Ming Wang, Jin Li, Jinmiao Fu, Xin Shen, Jingbo Shang. and Julian McAuley. Text is all you need: Learning language representations for sequential recommendation. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 1258- 1267, 2023.
12

---

Published as a conference paper at ICLR 2026
Jiachang Liu, Dinghan Shen, Yizhe Zhang. Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Eneko Agirr,, Marianaa Apidianaki, and Ivan Vulié (eds.). Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop Knowledge Extraction and Integration for Deep Learning Architectures, pp. 100-114, 11,, Dublin, Ireland and Online, May 2022. Association for Computational L Linguisties. doi: 10. 10.18633/v1/ 2022.declio- 1.10. URL
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng, Diyi Yang, Denny Zhou, and Andrew M. Dai. Best practices and lessons learned on syn- thetic data. In First Conference on Language Modeling, 2024.
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and downstream tasks for graph neural networks. Proceedings of the ACM Web Conference 2023, 2023.
Jieyi Long. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.06291 2023.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems, 36, 2024.
Yu Meng, Jiaxin Huang, Yu Zhang, and Jiawei Han. Generating training data with language models: Towards zero-shot language understanding. Advances in Neural Information Processing Systems, 35:462-455 477, 2022.
Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. Seallms - large language models for southeast asia, 2024. URL htLhttps::/ http/vixvvv.oo// org/ abs 20122...02077
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2022a. URL https: org/ abs/ 2203 02155
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022b.
Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, and Natasha Jaques. Personalizing reinforcement learning from human feedback with variational preference learning. arXiv preprint 10070 2024. ARthhtts:///axiii org/ 3b/02408.02055
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982-3992, Hong Kong, China, November 2019.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 7370-7392, Bangkok, Thailand, August 2024. Asso-
ciation for Computational Linguistics. doi: 10. 18.18653/v 10.11865/11/224..el 12024.acl-long.399 URL https:
c.clattholooy..rr//22222 acl-0022..ccaloooo long. 3997
Patrick Schober, Christa Boer, and Lothar A Schwarte. Correlation coefficients: appropriate use and interpretation. Anesthesia & analgesia, 1 126(5):1763-1766) 2018.
13

---

Published as a conference paper at ICLR 2026
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. Large language models are inconsistent and biased evaluators. arXiv preprint arXiv:2405.01724 2024.
Zhaoxuan Tan, Qingkai Zeng, Yijun Tian, Zheyuan Liu, Bing Yin, and Meng Jiang. Democra- tizing large language models via personalized parameter-efficient fine-tuning. arXiv preprint arXiv:2402.04401, 2024a.
Zhen Tan, Dawei Li, Song Wang, Alimohammad Beigi, Bohan Jiang, Amrita Bhattacharjee, Man- sooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. Large language models for data annotation: A survey. arXiv preprint arXiv:2402. 13446, 2024b.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cearuction-follovee llama model.
com/ htt/:/ttsuu-te http:/aabbsaaffrd__aaaalaaa
2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 2023.
Zhuozhuo Tu, Cheng Chen, and Yuxuan Du Rrinf: Efficient influence function estimation via ridge regression for large language models and text-to-image diffusion models. In Proceedings of the 2025 Conference on Empirical Methods Natural Language Processing, pp. 18505-18518, 2025.
Jesper E. van Engelen and Holger H. Hoos. A survey on semi-supervised learning. Machine Learning, 109(2):333 440, 2020. doi: 10. 058.0977. URL https://link. springes com art arttacieieeli 1007 000.000/11119401---0000
Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. Minilm: Deep self- attention distillation for aask-agn compression of pre-trained transformers. Advances in Neu- ral Information Processing Systems, 1-5777--7785 2020.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- cry, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203. 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, and Quoc V Le. H. chi, sharan narang, aakanksha chowdhery, and denny zhou, self-consistency improves chain of thought reasoning language models. In The Eleventh International Conference on Learning Representations, volume 1, 2023.
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long Le, Jin Miao, Zizhao Zhang, Chen- Yu Lee, and Tomas Pfister. Codeclm: Aligning language models with tailored synthetic data. In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 3712-3729, 2024.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.0160 2021.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
Ka Wong, Praveen Paritosh, and Lora Aroyo. Cross-replication reliability-an empirical approach to interpreting inter-rater reliability. In Proceedings of the 59th Annual Meeting the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 2053-7065, 2021.
14

---

Published as a conference paper at ICLR 2026
Siyuan Wu, Yue Huang, Chujie Gao, Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xian- gliang Zhang, Jianfeng Gao, Chaowei Xiao, and Lichao Sun. Unigen: A unified framework for textual dataset generation using large language models, 2024. URL abs/2406. 1194666.86666 8966
Qiaoliang Xiang, Qi Mao, Kian Ming Chai, Hai Leong Chieu, Ivor Tsang, and Zhendong Zhao. A split-merge framework for comparing clusterings. arXiv preprint arXiv: 64755, 2012.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms. arXiv preprint arXiv:2306. I 13063, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https:
//openreview.o htt.://iww...eettooomomo--------
Jiacheng Ye, Jiahui Gao, Qintong Li, Hang Xu, Jiangtao Feng, Zhiyong Wu, Tao Yu, and Ling- peng Kong. Zerogen: Efficient zero-shot learning via dataset generation. arXiv preprint arXiv:2202.07922. 2022.
Hanlei Zhang, Hua Xu, Ting-En Lin, and Rui Lyu. Discovering new intents with deep aligned clustering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14375, 2021.
Xinlu Zhang, Chenxin Tian, Xianjun Yang. Lichang Chen, Zekun Li, and Linda Ruth Petzold. Alpacareiinstruction-ue
large language models for medical application, 2023a.
Yuwei Zhang, Haode Zhang, Li-Ming Zham, Xiao-Ming Wu, and Albert Lam. New intent discovery with pre-training and contrastive learning. arXiv preprint arXiv:2205. 12914, 2022.
Yuwei Zhang, Zihan Wang, and Jingbo Shang. Clusterllm: Large language models guide for text clustering. In The 2023 Conference on Empirical Methods in Natural Language Processing, 2023b.
Kaitlyn Zhou, Jena D Hwang, Xiang Ren, and Maarten Sap. Relying on the unreliable: The impact of language models' reluctance to express uncertainty. arXiv preprint arXiv:2401.06730, 2024.
Jingge Zhu. Semi-supervised learning: the case when unlabeled data is equally useful. In Proceed- ings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), vol- ume 124 of PMLR, 2020. URL https: https::/prrocedings mlr prwss/w124 zhu20b. html
15

---

Published as a conference paper at ICLR 2026
APPENDIX
Appendix Overview.
Section
Content Summary
A
Notation Summary
B
Experimental Sctup
C
Pearson Correlation Test for CAI Scores and LLM Accuracy (C)
Statistical Inference [.II]
C.2
Before Applying the Method
C.3
After Applying the Method
Dataset Descriptions
D
D.I
Benchmark Datasets
D.1
R
Baselines
F
F
Evaluation Metries
F.I
Reference Free Evaluation Metric (CAI)
F.I
Comparison with Traditional Evaluation Metries
F2
G
Additional Experimental Details G
G.1 Inverse-Comsistent (IC) Ratio G.1
Why LLM--peciallized Model Collaboration Essential for HCC (H
H.I Role of the Specialized Model in HCC
H.O.1
Sensitivity Analysis of K Across Different Specialized Models
Additional Ablation Studies (Llama-3-8B-Insteee [
J.1
HCC Under Weaker (BERT) Stronger (Llama-3-8B) Models J.I
HCC Across Embedding Backbones and Smaller Language Models
J.3
CAI Ratio Across Backbones (Before vs. After HCC) J.3
J.4
Statistical Validation of CAI Across Encoders and LLMS
0.4
J.5
Running Time and Token Cost (J.5)
J.6
HCC Runtime for Co-Alignment J.6
1.7
Two-Round Correction and Improvement
K
Ablation Studies CharGPT-40 Mini
K
K.1
Failure Case of CAI K.I
K.2
Running Time and Token Cost
K.2
K.3
HCC Runtime for Co-Alignment K..3
K.4
Two-Round Correction and CAI Improvement K.4
Ablation Studies CharGPT-3.5 Turbo
I..1
Failure Case of CAT [T.1
L.2
Token Usage and Running Time L.2
L.3
HCC Runtime for Co- Alignment
L.4
Two-Round Correction and CAI Improvement
L.4
Additional Imbalance Ablations (1%, 5%, 10% User Preference)
N
Q
Overall Reflection on CAI N
O
Generalization
O
P
Model Selection Studies
Q
Handling Conflicting User Preferences
R
Algorithm Table (R)
S
Prompt Instructions
S
S.1
Prompt Instruction (Without Preference Context) S.I
S.2
Prompt Instruction (With Preference Context) SS.2
T
L-SNE Visualization for Clustering All Datasets (ChatGPT-49 Mini)
U
Usage of the Large Language Model (LLM) U
16

---

Published as a conference paper at ICLR 2026
A NOTATION SUMMARY
Summary of main notation used throughout the paper.
Symbol
Description
X c Rd
Input space of textual instances.
2j
The i-th input instance.
Du { r=
Unlabelled (semi- superviseevise) corpus.
Des
= fi
Test corpus used for ovaluation.
y (l.....(]
Discrete label space with C' classes.
H1 (1i ii)
Small user-preference seed set.
spec-spee embedding model (e.g., MiniLM, BERT). Large language model (LLM) annotator.
T
6 = S(x))
Sentence embedding Jzy
9 (5)
Annotation assigned by specialised model Zero-shot annotation assigned by LLM T.
Single-shot annotation assigned LLM T'
Ds { {1
Datasct annotated by specialised model.
Dr = ((Ei, Vi T
Zero-shot LLM-annotated dataset.
Dr = ((Ei, }}
Single-shot LLM annotated dataset.
C
Set of consistent samples (inter-model agreement).
I
Set of inconsistent samples fnter-mode disagreement).
C= (ei E Da: T(x) S(a))
Formal definition of consistent set.
= (2i E Du: :(:T( # ((.))
Formal definition inconsistent set.
No = |CI
Number of consistent samples.
N1 !T|
Number of inconsistent samples.
CAI(D.;T.S)
Consistent-and Inconsistent Ratio.
Nc
CAI N₁ +
Reference-free reliability metric.
CI
Subset of I corrected in Round 1.
Subset of 7 remaining inconsistent after Round 1. Inconsistent set after Round reassignment.
(1)
II(I)
Hard inconsistent samples corrected Round Fully aligned inconsistent set.
( CI
D (Rmal)
Final corrected dataset: C CUCTU TZ()
K
Number of nearest neighbours used MV-VTES.
DUH
Reference set for embedding based correction.
TopK(a Ac)
Operator selecting K ncarest neighbours in embedding space. Label from retrieved neighbours. Frequency of label among neighbours. Majority-vote refined annotation.
A (1L ... VR
Nh EØ=1 [ a)
④ arg maNac A fhe
17

---

Published as a conference paper at ICLR 2026
B EXPERIMENTAL SETUP
We evaluate the CAI Ratio on multiple LLMS-GGT-3.5-TubbS GPT-4o-Mini, Google Gemini 1.5 Flash (Model Selection Task), and span diverse range of domains and task types, including intent classification, topic modeling, sentiment/emotion analysis, and semi-supervised intent discovery (Zhang et al. 2021 2022). The annotation practices follow Zhang et al. (003bb).
across ten textual datasets. These datasets
Intent Classification.
Banking77 Casanueva et al.
2020
CLINC150 Larson et al. 2019a)
MTOP (Li et al.
2020
Massive (Intent) Larson et al. 2019b
FitzGerald et al. 2022)
Topic and Question Classification.
StackExchange al. 2021
Reddit (Geigle al. 2021
Emotion and Sentiment.
GoEmotioon (Model Selection Task)
Relation and Entity Classification.
FewRel-Nat (Han et al. 2018)
FewNerd-NNt (Model Selection Task)
C PEARSON CORRELATION TEST FOR CAI SCORES AND LLMS ACCURACY
C. STATISTICAL INFERENCE
We have performed a Pearson correlation, the correlation coefficient r calculated as:
E () 9)
(z
g)
where x; symbolises the CAI ratios. Y/ denotes the LLM annotation accuracies. 2 and g are the average mean of x; and y/, accordingly. n is the number of samples we have used for evaluation. To assess the statistical significance, we use a hypothesis test for the correlation coefficient, calculating t-statistic (Schober et al. 2018):
The P-value is then calculated from the t-distribution with degrees of freedom.
C.2 BEFORE APPLYING THE METHOD
The Pearson correlation coefficient is 0.805, indicating strong positive linear relationship between CAI and Accuracy. The p-value 0.005, which is statistically significant (below the typical thresh- old of 0.05). This implies that the positive correlation between the CAI ratio and Accuracy before and after applying our method is not a random event, and higher CAI scores are associated with higher Accuracy. The p-vulue is 0.00093, which is statistically significant (below the typical thresh- old of 0.05) for the additional datasets. The p-value is 0.014399, which is statistically significant all datasets.
(below the typical threshold of 0.05) for -aaa---lla------iittttt
18

---

Published as a conference paper at ICLR 2026
Table 4: Pearson correlation results for CAI and accuracy, before and after our proposed method on GPT-3.5-Ture and GPT-4 Mini, before and after our proposed method.
Metric
Correlation
Before
0,805
0.005 0.00033
Aller
0.903
Table Pearson correlation results for CAI and accuracy for additional datasets on GPT-3.5-Turbo and GPT-4 Mini, before and after our proposed method.
Metric
Pearson Correlation
P-value
Before
0.874
0.000937
After
0.852
0.00175
Table 6: Pearson correlation results for CAI and accuracy Sita-llama---B-innttte before and after our proposed method.
Metric
Pearson Correlation
P-value
0.0144
After
0.918
0.00129
C.3 AFTER APPLYING THE METHOD
stronger positive correlation between CAI and Accuracy after applying the method. A larger CAI ratio and higher annotation produced by LLMs extremely statistically significant, according the p-value of 0.00035. This implies that the relationship between CAI and Accuracy is even more evident after using the approach, showing more linear relationship where increases in CAI are more directly correlated with increases in Accuracy. In both stages (Before and After applying the method). the results display statistically significant correlations (p < 0.05), showing strong positive relationships between CAI scores and LLM accuracy. Tables o that all the v-values of the Pearson correlation are statistically significant.
The Pearson correlation coefficient is 0.903, showing
D DATASET
Task
Name
data (Testing)
data (Training)
# classes
Bank77
3,080
10,003
77
CLINC (I)
4,500
15,000
150
Intent
MTOP (l)
4,386
15,638
102
Massive
2,974
11,510
59
Type
Few Rel
4,480
40,320
64
StackEx
4,156
50,000
121
Topic
Reddit
3,217
50,000
50
Domain
Massive Scenario
2,974
11,514
Table 7: Summary of Benchmark Datasets.
D.1 BENCHMARK DATASETS
We evaluate our work on a series of open-source textual datasets spanning diverse domains, includ- ing Bank77, CLINC (Intent), MTOP (Intent), Massive (Intent), StackExchange and Reddit (Topic) (Geigle et al. 2021). and Few Rel Nat (Type). We also utilize the Massive Intent dataset, with annotation practice following (Zhang et al. 2023b). Bank77 (Casanueva et al., 2020) a banking dataset that focuses fine-grained intent classification within single domain. CLINC (I), Mas-
sive (I), and MTOP (I) are intent-based datasets (denoted as "T") (Larson et al. 2019bb FitzGerald et al. 2022 Li et al. 2020). Intent discovery (Zhang al. 2021 2022) explores unknown intents in semi-suue Semi-supervised utterance datasets. Each dataset is available in small-scale (testing) and large-scale (training) versions, and use i.i.d. user preference samples from the training set. Since
19

---

Published as a conference paper at ICLR 2026
CAI Score vs LLM Accuracy (Before Our Method)
CAI Score VS LLM Accuracy (After Our Method)
LLM Accuracy (%)
Chorght few Cha
Accuracy (%)
CharsPt few Chor
8
8
ChenSPT N
Chah M
massive intant ChsIdPT
massive intant ChaidPT infent ChuidP74 Mes Line (PI 0.705)
8
intent CheldPT Meni
8
Line (er 0.213)
CAI Score
CAI Score
(a) GPT-3.5 Turbo and GPT-4o Mini
(b) GPT-3.5 Turbo and GPT-4o Mini
Figure The above analysis shows the correlation between LLM annotation accuracy and the munsistent-anddlncoonssttee (CAI) ratio. We also conducted statistical tests to assess the significance of this correlation. We collected the CAI ratios for (LLMs 3.5 Turbo and Specialised Model ) and (LLMs 4.0 Mini and Specialised Model across the datasets Reddit, few rel nat and massive intent. Using these data, we calculated the Pearson correlation coefficients between the LLM annotation accuracies and CAI ratios and computed the associated P-values (P value for After: 0.036) and (P value for Before: 0.014) to determine the statistical significance of the observed correlations.
CAI Score vs LLM Accuracy (Before Our Method)
CAI Score vs LLM Accuracy
(After Our Method)
tie Zuckknchunge
Chec Nossan_tooeere
LLM Accuracy 196)
6
LLM Accuracy (%)
Stackltrc ange Burking??
Suriing3
3
Mushhs beae
Moalio hart Ro lin doe 6
Fi tne (or B.i0e)
CAl Before
CAI After
(a) Llama-8B ---trructt
(b) Jlama-8B-Instrus
Figure 6: The above analysis shows the correlation between LLM annotation accuracy and the Consistent-and-Inee Incoonsssennnnttnnn (CAI) ratio. We also conducted statistical tests to assess the significance of this correlation. We collected the CAI ratios for (LLMs 3.5 Turbo and Specialised Model and (LLMs 4.0 Mini and Specialised Model ) across the datasets CLINC, Massive Scenario, MТOP Intent, Stack Exchange, and Banking77, Reddit, Few Rel Nat, Massive Intent. Using these data, we calculated the Pearson correlation coefficients between the LLM annotation accuracies and CAI ratios and computed the associated P-values (P value for After: 0.014) and (P value for Before: 0,00129) to determine the statistical significance of the observed correlations.
working in an semi-supervised textual data setting, we directly use the small-scale for testing. Detailed statistics of the datasets are shown in Table
E BASELINES
To rigorously assess proposed framework, compare against diverse set of strong baselines spanning specialized models, prompting methods, and clustering based annotation strategies. Each baseline represents distinct family of annotation approaches relevant to the semi-supervised setting.
Specialised Model Only. pre-trained fask-specialised model assigns labels to unlabeled data us- ing preference-based annotation scheme and a small set of user-provided samples. This baseline measures the standalone utility of the specialised model without assistance from LLMs or consis- tency mechanisms. The specialised model relies our proposed semantic clustering approach to organise examples into coherent intent-level groups and to assign each unlabeled instance a label that reflects the user-provided preference samples.
20

---

Published as a conference paper at ICLR 2026
LLMs Only (Without Intent from Specialised). We evaluate GPT-3.5 and GPT-40 mini in zero-shot setting, prompting them directly to generate category labels. While LLMs offer a scalable and cost-effective annotation mechanism, their outputs prone to inconsistency, hallucination, or drift from user preferences in the absence of supervision or demonstrations. In the L.Mss Only setting, the model is prompted using semantic hatch clustering, where examples come from the same latent category but no explicit label is provided. This offers structure but not supervision.
Specialised Model + LLMS (With Intent from Specialised). This variant employs our proposed Majority Voting via Top-Nearest Embedding Similarity using the task-specialised model to gener- ate pseudo-labels for unlabeled samples. These pseudo-labeled instances then serve as in-context demonstrations for guiding LLM annotation. This hybrid setup improves alignment while retaining annotation efficiency. In contrast, the Specialised Model + LLMs (Ours) setting augments the same clustered batch with an explicit intent label. This converts the prompt from a purely unsupervised, similarity driven signal into supervised alignment cue, enabling the LLM to ground its predictions in both semantic consistency and explicit task semantics.
HCC w/o Co-Alignment. This ablation examines simplified variant of our Heterogeneous Con- sistency Co-ACli Co-Aligmment (HCC) framework. We exclude inconsistent samples and distill knowledge only from high-come cegssconsittency examples into pretrained BERT-base model. While this improves annotation precision, it lacks the full co-alignment loop present in HCC, making it less robust to preference drift.
Clustering Baseline. We compare against Zhang al. (2023b). state-of-the-att clustering ap- proach for few-shot annotation, where labels are propagated by embedding similarity. Unlike our method, lacks preference modeling and relies heavily on embedding space separability, Our Top- Nearest Majority Voting introduces new clustering method tailored to the semi-supervised regime.
Prompt-Based Approaches. We evaluate several representative prompting methods, all using KATE (Liu al. 2022) to select top-k similar demonstrations from user-labeled data:
2022) selects the most frequent response across multiple sampled reasoning paths, assuming the L.M is sufficiently competent. However, it incurs high compute cost and often reflects pretraining bias rather than user-specific intent.
Seff-Consistency (Wang al.
Chain-of-Thouue (CoT) Prompting (Wei et 2022 guides the model reason step- by-step via structured exemplars, improving interpretability and answer accuracy.
Few-Shot In-Context Learning (Brown et al. 2020) provides handful of labeled exam- ples in the prompt to elicit more accurate model behavior, yet its effectiveness is sensitive to demonstration quality.
(Madaan et al. 2024 iteratively improves model outputs through refinement steps and feedback, though its performance relies on LLM strength and auxiliary tool sup- port.
F EVALUATION METRICS
We evaluate our method and the baselines based on two metrics: (1) Annotation Accuracy; (2) our proposed CA/ ratio. Annotation Accuracy evaluates the correctness of LLM generated annotations, while the CAI Ratio measures the model's ability to correct inconsistencies. higher CAI Ratio after applying method indicates improved annotation and effective co-alignment.
F.1 REFERENCE-FREE EVALUATION METRIC
Intuitively, inter-rater reliability (IRR) metrics such as Cohen's kappa and Krippendorff's alpha, which are widely used for assessing the reliability of crowdsourced annotations, may seem suffi- cient. However, these metrics assume that annotators are equally competent and operate under fixed candidate label set, the condition that are hardly satisfied in reality. IRR becomes unstable when the class distribution is skewed, when annotators systematically omit certain classes, under
21

---

Published as a conference paper at ICLR 2026
marginal drift caused by heterogencous annotator competence, in open-set scenarios where shared label set exists Wong et al. (202T). Consequently. IRR is less reliable for personalized or sub- jective tasks, where cultural training differences amplify annotator variance and bias the results. By contrast, the Consistent-and-Inconssi eiference-free metric designed quantify annotation reliability by leveraging heterogeneous consis- lency signals nikelihood-deriver and embedding agreement) without relying on references or oracle annotations.
(CAI) ratio is label-set agnostic and represents the first
F.2 COMPARISON WITH TRADITIONAL EVALUATION METRICS
In Table which shows a comparison with CAI ratio and Accuracy, Precision/Recall and FI-score in term of ground-trute label, robustness to Data Drift, and tracking annotation alignment over time.
Metric
Ground- Labels?
Data Drift?
Tracks Annotation Alignment Over Time?
Accuracy
X
X
Precision/Recall
X
X
Fl F1-score
X
X
CAI Ratio
X
/
/
Table Comparison of Traditional Metrics and CAI Ratio
G EXPERIMENTAL DETAILS
The top-k selection and proportions of consistent and user-preference samples are as follows. For CLINC and Massive Scenario, 'top-k' is set 5, with 'proportion' 0.2. For MTOP Intent, 'pro- portion' is set to 0.8, and 'top-k' is updated to 15 after printing the current value. In StackExchange, 'top-k' set to 5 and 'proportion'
1, while in Banking77, 'top-k' set to 3 and 'proportion' 0.2. In massive intent, 'top-k" is 20 and "proportion" is 0.5), proportion=0.2, and few real nat has top-k=30, and proportion is In 'Reddit', 'top-k' is set to 7, and the proportion 0.2. All tests are done with two random seeds with temperature parameters (0.5 and 1) for user preference samples, task-specialised model-assigee assiined annotation, and LLMs with and without task-specialised model annotations. We have methods and baselines with random seeds.
G.1 INVERSE CONSISTENT (IC) RATIO
The number of samples per class required for human annotation based termined by our Inverse Consistent (IC) ratio equation G.I For user-preference samples. The n denotes the total size of the the consistent sample where M n,, and be the number of classes. The parameter represents the proportion of samples to be selected and is set to 5% (i.e., 0.05). In our experiment, we do not use all the identified consistent samples. The proportion of consistent samples used for co-alignment is determined by the IC ratio. Let n. be the number of consistent samples, so M ne represents the size of the consistent sample selection. If the CAI ratio is greater than 0.5 (i.e., the number of consistent samples exceeds inconsistent ones), the value of p will be reduced to use fewer consistent samples. the CAI ratio is less than 0.4, p is set to 1 (i.e., 100%) since more consistent samples are needed for co-alignment. The formula for the
preferences is de-
Inverse Consistent (IC) ratio defined as IC MMx
H WHY GENERAL LLM AND SPECIFCC MODEL COLLABORATION IS ESSENTIAL FOR HETEROGENEOUS. CENOOGGENEEOSSCCCSSSTEENNC Co- CO-ALIGNMENT
In semi-supervised learning tasks that rely on preferences, the key challenge lies in evaluat- ing --eeereeertted annotations and enabling mechanisms for co-alignment- especially when the competency of general LLMs is uncertain and external knowledge is available. To address this,
22

---

Published as a conference paper at ICLR 2026
propose HCC, co-alignment framework that facilitates both self-assessment and refinement of LLM annotations through collaboration with a aask-specialized model.
HCC introduces the Consistent ieriistent-ann---e and-Inconsistent (CAI) Ratio. which quantifies agreements and dis- agreements between models to identify consistent and inconsistent samples. This iterative refine- ment process improves the performance of both the specialized model and the general LLM. To validate our approach, we experiment with Meta-8B Instruct, a lightweight LLM representing a low-competency model, and show that its collaboration with a task-specialized model enhances an- notation robustness even under noisy conditions.
More broadly, given semi-supervissd learning task where LLM competency is unknown and no external knowledge exists, HCC addresses the fundamental question: How can we evaluate and enable co-alignment for such datasets? We answer this through HCC, self-supervised strategy that allows self- ceff-correction and self-evaluation of LLM-geeeratte annotations. By combining the CAT ratio with task sss-specialized model collaboration, HCC systematically improves annotation quality across both models.
H.0.1 THE ROLE OF SPECIALISED MODEL IN HETTEROGENEOUS-CONSSSEE CO-ALIGNMENT
The inclusion the specialised model vital as provides a safeguard against underperformance by the LLM. Additionally, the task tusk-specialised model serves as reference point for "course track- ing." meaning that it allows us to monitor and guide the annotation process by comparing the task- specialised model's output with the general LLM's output. This approach is particularly evident in our experiments where the Meta-SB Instruct model, acting as a low-competency "LLM," demon- strated suboptimal performance The task-specialised model addresses this issue by collaborating with the general LLM to itera- tively refine annotations. This process ensures the framework's 's robustness, even when the general LLM lacks competency in specific tasks. We justify the necessity of the task-specialised model through experimental analysís (see Section & 4.3.1 and Table 4). These results show that our pro- posed HCC framework consistently outperforms baseline methods, even when paired with low- competency LLMs such as the Llama 8B Instruct model (Touvron al. 2023). This demonstrates the resilience of HCC and the critical role of the task-specialised model in enhancing performance across diverse LLM configurations. Moreover, recent studies (Zhou al. 2024] Xiong et al., 2023) highlight the inherent challenges of relying solely on LLMs, particularly their tendencies toward overconfidence and reluctance to express uncertainty. These findings further validate the inclusion of task-specialised model to mitigate such limitations.
most of the eight datasets, indicated by its low CAI scores.
Table 9; Meta-Llama 3-8B Instruct (open-source lightweight LLM): Annotation accuracy com- parison in percentages with standard deviations across different datasets for the specialised model LLMs without annotations from the task-specialised model, and LLMs with annotations from the lask-specialised model. The highest accuracy for each dataset is highlighted.
Datasets
Specialised Model
LLM Only
Specialised
HCC
w/o
HCC Corr
w/ CAT Ratio 1 (Be-
(Llama-3- SB-Instruct)
LLM
Corr
fore >> Afler)
Cline
79.01 2100
32.49 25.73
69.40
63.41 13.19
82.43
0.56=34.33
Massive Massive.Scenario
75.55 78
43.52 41.5
66. 2098
70.06 41.12
78.13 .0.34
0.67=$4.8
Mtop Intent
32.49.255
34.17 45.00
48.23 2025
66.39 20.00
63.39 2147
0.35 1
StackExchange
32,27
11.02
26.26
16.03
38,88
0,23=>0.53
Banking?7
1393=183
33,06 #1.90
69.66 21.74
64.29 41.24
77.71
0.68 +4,20
Reddit
31.73
36.31 40.97
46.00 =251
40.29
58.81 2022
0.33 1.38
Rel
35.35 2003
40.36
30.07 44.45
31.80 40.54
42.92 2008
0.13-+0.85
Massive Intent
=1.00
45.41
56.03
67.75
0.73
23

---

Published as a conference paper at ICLR 2026
Table 10: Performance Comparison of HCC: The table shows the performance of HCC compared to "Only Specialised Model" and "Only LLMs," with improvements highlighted.
Dalasets
Specialised (Ours)
Model
(Llama-3-88-
Improvement Specialised
Improvement LLM (%)
Instruct)
(Ours)
Model
(%)
Cline
79.01 s1.06
32.49 26.73
82.43 s020
+3.42
+49.54
Mussive_Seemnn Scenario
75.55
43.52
4071
Mtop Intent
52.49
34.17
63.39
+10.0
+29.22
StackExchange
32.27 unse
11.02 42.06
38,88 wn.1
+6.61
+27.86
Banking77
73.93 2135
33.06 2112
77.71 an:s
+3.78
+44.65
Reddit
51.73
36.31
58.81
+22.50
Few Rel Nat
35.35 wnsn
14.25 40/6
42.92 40.0
+7.57
+28.67
Massive Intent
61.80 D1
45.41 +006
67.75 10.13
+3.95
+22.34
I SENSITIVITY ANALYSIS OF K ACROSS DIFFERENT SPECIALISED MODELS
To assess the robustness of HHC, we extend ablations to three diverse sentence encoders (spe- cialised models) and a small language model (serving as a weak model) collaborating with Llama-3- 8B-Instruct (serving strong model). We have also included the parameter sizes of all specialised models in Table
E5-base (Sentence Encoder)-E5
GTE-small (Sentence Encoder)-GTE
BGE-small-en-vl.. (Sentence Encoder)-BGE
BERT-Base-uncaser (Small Language Model)-BERT
Encoders and Language Model
Parameters
Role in Ablation
MiniL-M-L6-v
22.7M Params
Small, General- SBERT Strong, Retrieval-focuset
E5-base
~110 Params
GTE-small
33.44 Params
Small, High-quality Encoder
SGGE-sall-enn-vl..
33344 Params
Small, Retrieval serrevaal-optimzzed
BERT- Baee--Bas-unesed
~10M Params
Small. Language Model
Table 11: Embedding backbones and the weak language model used in the robustness ablation.
Specialized Model Accuracy Banking77
Specialized Model Accuracy CLINC
Specialized Model Accuracy FewRel Nlat
Specialized Model Accuracy MTOP Intent
Specialized Model Accuracy Massive Intent
Specialized Model Accuracy Messive Scenario
Specialized Model Accuracy
Specialized Model Accuracy StackExchange
Reddit
Figure 7: Accuracy of specialised models across datasets for different values of K. Performance is compared for five backbone encoders: BERT, BGE, MiniLM, E5, and GTE.
We have conducted K sensitivity analysis as shown Table 12 with Specialised Models (We added
ablations using E5-base, GTE-small, and BGE-small in addition to the standard BERT baseline.) K Ablation. We implemented full sensitivity study with K E {3, 5, 10) across datasets. Results show minimal deviation across K, with 3 or K consistently performing best for fine-grained tasks.
24

---

Published as a conference paper at ICLR 2026
Dataset
Backhone
K=3
K=5
K=7
K=10
0.796
0.002
0.788 + 0.002
0.780
0.006
0.767 +: 0.014
all MiniLM L6
0.741
0.006
0.727
1 0.003
0.718
0.004
0.702
1: 0.004
Banking?7
bert uncased
0.372
0.018
0.025
0.129 4: 0.020
intfloat-eS- ---ba------ae
0.715 0.000
0.700 = 0.000
0,702
0.007
0.697
= 0.004
thenlper-gte-small
0.761
0.007
0.742 = 0.000
0.006 0.732
0.726 = 0.011
0.833 = 0.002
0.830 = 0.005
0.825
0.002
0.817 = 0.004
all-MiniLMM-L6-22
0.794
0.787 0.007
0.782 0.001
CLINC
bert-buse uncased
0.557 0.005
0.533 = 0.008
0.515 = 0.007
0.493 + 0.002
intiloal base
0.007
0.792 1: 0.001
0.010
0.788 1: 0.015
seeniper-gie-saan
0.808 0.007
0.009 0.803
0.795
0.007
0.792 = 0,002
0.375 0.003
0.372 0.001
0.370
0.000
0.367 + 0.003
all- Ml-MMinLLM-L6-22
0.350 = 0.003
0,346 = 0.002
0.342 = 0.003
0,341 = 0.000
VewRel-Nat
uncased
0.012
0.418
intflnat-e5-baee ---see
0.408 0.006
0.406 = 0.009
0.405 0.008
0.402 - 0.009
thenlper-gte-small
0.375 0.012
0.375 = 0.012
0.373 = 0,014
0.371 0.015
BAAI-bge-small-n---
0.557 0.012
0.536 = 0.011
0511 0.020
0.508 + 0.013
Ml-MMini.M--L-v22
0.535
0.526 0.016
0,520
0.518 0.025
MTOP Intent
bet-bass-uncased
0.503 = 0.015
0.483 = 0.025
0.451 = 0.020
0,437 = 0.031
intflost-e5-bsse
0.008
0,855 = 0.006
0,400 0.008
0.490 + 0.012
thenlper-gte-small
0.500 = 0.014
0.015 0,488
0.023 0.473
0.471 0.036
BAA--gg-small-anvvi.
0.692 0.006
0.683
0.013
0.655
0.019
0.657 = 0.035
all-MiniLM--6--
0.606 0.014
0,587 = 0.011
0.574 = 0.015
0,566 = 0.011
Massive
0.390
0.350 0.028
intiloal e5 base
0.614 0.002
0,606 1 0.002
0.001 0.599
0,582 1 0.005
hhenlpr-gtt--sall
0.007
0.616 = 0.006
0.002
0.604 0.001
BAAL bge small en.
0.782 0.025
0.779 1: 0.026
0.776
0.027
0.770 1 0.030
all-Mini.MM-AA
0.753 0.014
0.752 0.013
0.748
0.010
0.741 0.009
Massive Scenario
bert buse uncased
0.601
0.023
0.591 1 0.029
0.030 0.580
0,570 1 0.032
intflrat-eS-buse thenlper-gte-small
0.027
0.711 = 0.033
0.701 + 0.032
0.752 = 0.022
0.015 0.750
0.016 0.749
0.747 = 0.015
BAAL Algee-ge--ae -nal--mall-n--1.55
0.512 0:005
0.510 = 0.004
0.506
0.006
0.501 + 0.005
M--M--
0.515 0.002
0.517 = 0.001
0.515 = 0.003
0,513 - 0.000
Reddit
base-uncceedd
0.005
0,304 0.008
inilloal eS base thenlper gle small
0.512 0.006
0.512 + 0.001
0.002 0.505
0,503 + 0.001
+ 0.004
0.002
BAAT-bge-snnd 1--l--ge-amll-----115
0.376 0.006
0.367 = 0.011
0.359
0.009
0.351 + 0.007
all-MiniLMM-L6-22
0.314 0.007
0.297
0.011
StackExchange
bert suse-uncased
0.277 0.005
0,268 = 0.003
0.263
0.006
0,258 + 0.009
intfloal- buse thenlper-gter -eea-ll
0.004
0.375 0.006
0.361
0.007
0.345 0.006
0.399 0.013
0.395 = 0.020
0.388 0.019
0.379 + 0.020
Table 12: Mcan + standard deviation of specialised model accuracy for different values of K across datasets and backbone models.
J ADDITIONAL ABLATION STUDIES BASED ON - LARGE LANGUAGE MODEL
J.1 HCC UNDER WEAKER (BERT-BASE-UNCASEE vs. STRONGER LANGUAGE MODEL. (LLAMA-3-88 INSTRUCT) REGIMES
To demonstrate that Heterogeneous Consistency Co-Alignment (HCC) remains effective across both weaker and stronger language model regimes, we adopt BERT BERT-Base Unccaeed as weak spe- cialised model and 1-1ama-3-a0 8B Instruct as a strong LLM. As shown in Appendix Tables I3 and 14 HCC enables the open-source Llama-3-8B Instruct even when paired with the weaker BERT backbone- to outperform GPT-4o Mini on five datasets and GPT-3.5 Turbo on seven datasets. This highlights HCC's ability to boost weaker specialised models while narrowing the per- formance gap between open-source and proprietary LLMs. In this setting, bert seset--ase-uncee used solely for the initial annotation assignment. However, using BERT for the divide-conquer co-alignment stage degrades performance due to its weaker semantic representations. To address this, we adopt a hybrid encoder strategy: during the co-alignment phase, BERT replaced with all-MiniLMM-L--2. substantially stronger SBERT encoder.
This hybrid design produces more coherent semantic clusters and significantly improves HCC per- formance (Tables 13 and 14), often surpassing both GPT-4o Mini and GPT-3.5 Turbo. Thus, BERT
25

---

Published as a conference paper at ICLR 2026
is used initialise label assignments, while MiniLM provides the semantic granularity required for reliable divide conquer co-alignment.
Table 13: Performance Comparison: HCC (R2 with BERT-Base-Uncasee vs. Closed-Source Models. HCC uses only a weak BERT encoder, yet still outperforms GPT models on most datasets. (K=3)
Dulasel
ICC (BERT)
GPT-4o Mini
GPT:33 Turbu
Outcome
CLINC
0.8240
0.8144
0.6658
Beals Both
Massive Scenario
0.7461
0.6683
0.6089
Beats Both
Banking?7
0.7162
Beats
Reddit
0.5813
0.5740
0.5112
Bents Both
0.4205
MTop Intent
0.6562
0.7503
0.6495
Beals GPTT-355 Only
Exchanx
GPT-33,5 Only
Mussäve Intent
0.6261
0.7693
0.7152
Lower than both
Table 14: Performance Comparison: HCC (R2 with SERRT-Bss--ncaasee vs. Closed-Source Models. HCC uses only weak BERT encoder, yet outperforms GPT models on most datasets. (K=10)
Dutaset
HCC (BERT)
GPT-46 Mini
GPT:3.5 Turbo
Outcome
CLINC
0.7958
GPT--3.5
Mussive Scenario
0.7576
0,6683
0.6088
Beals Both
Banking?7
0,7234
0.6512
0.6512
Beats Both
Reddit
0.4085
0,5740
0,5112
Lower than both
Rel-NeNat
0,4292
0.3287
Beats Both
MTop Intent
0.5857
0.7503
0.6495
Beats GPT-3.5 Only
StackExchunge
0.3898
0.5390
0.3010
Bents GPT-33.5 Only
Mussive Intent
0.6187
0.7693
0.7152
Lower than both
KEY FINDINGS:
1. On CLINC, Banking77, Massive Scenario, Reddit, and FewRel, the HCC framework is effective that it endows (standard BERT model Llama-3-8B) surpass the annota- tion quality of GPT-3.5 and GPT-4o Mini.
J.2 HCC ACROSS EMBEDDING BACKBONES AND SMALL LANGUAGE MODEL
The comparison of HCC across the best embedding backbones with ChatGPT-40 Mini and ChatGPT-3.5 Turbo is presented in Tables 18 and 17 Tables 19 and [0 further illustrate the per- formance of the HCC paradigm using different embedding backbones, where the best encoder is selected for each dataset in conjunction with Llama-3-8B. We compare this configuration against GPT-4o Mini and GPT-3.5 Turbo to highlight its robustness and competitiveness. Our ablation stud- ies in Section J. and Section J.2 show that HCC moderately robust even when the specialised model is weak. For instance, using BERT as the specialised encoder, HCC still surpasses both stan- dalone specialised models and standalone LLMs, regardless of whether clustering is performed with without intent information. This indicates that HCC compensate for imperfect cluster struc- tures produced by weaker encoders. At the same time, the results show that HCC benefits consis- tently from stronger embedding encoders. Models such E5, GTE, and BGE yield progressively higher accuracy, demonstrating that improvements in representation quality translate directly into more reliable clustering and, consequently, stronger co-alignment performance. Overall, under the standard K = 3 and k=10 settings, HCC remains stable across both weak and strong specialised mod- els, confirming that is broadly effective for categorical NLU annotation tasks while still offering additional gains when higher- encoders are available.
J.3
CAI RATIO ACROSS EMBEDDING BACKBONES (BEFORE/AFTER HCC)
Table 19 and 20 shows how the CAI Ratio behaves when pairing Llama-3-8B with four special-
ized embedding backbones varying capacities- BERT (weak), GTE-small (moderate), E5-base (strong), and BGE-small-en-115 (strong). For each dataset- configuration, we report the
26

---

Published as a conference paper at ICLR 2026
MCC Roundo Accuracy vs JDrtmntt banideg77)
Acourcy (Dnteet diinc
Round2 Accurncy va. fnn.st.rrt
AecurrAccuury mumbr, Piteni
H42 2 AADARR
Anearary mtep selee]
Round-22 Accurtry
Accuncy stadeenchrrpe
Figure 8: Accuracy of HCC (R2) across datasets for different values of K(3,10) Llama3-8 8 In- struct
is compared for four backbone encoders: BERT, BGE, E5, and GTE.
Table 15: Averaged performance over seeds 423 for K e {3), reported as mean + standard deviation for each PeatasttBBackbrnd pair.
Specialteed
bintinghT
small
0,3385 0.0000
0,045
0
0.0055
0.3101
0,7164
bunling?
bert unaeeaad-nneed
0.1804
0.0034
0,5810
0.0032
0.1700
0.6255
0.6050
0,6212
0.1575
0.0101
0.422-
0,066
0.3945 0.005
hinkingT
untoan tt-base
0.2218-:-00 0.0174
0.6277
0.0451
0,0000
0.0024
2.9100
0.2818
0.5355
0.00015
0.7584
0.155
0.714 & 1121012
bunling?7
thelpes atc snall
0.5185 0.0057
0.8010
0,0040
0,4700
0.0041
3,9160
0.1768
0.7763
2 0.0045
0.7747
0,0041
0.7614 =: 0.0075
untioa
0.3015
0.0050
0.0870
0.0066
0,0000
0.00900
0.5141
0.0041
08211
0.003
U.8011 U.2UT1
cline
thelper-dte ate small
0,2061
0.0071
0,7018
0.0196
0,5660
0.0073
4,1000
0,0086
0,8242
0.0033
0.3182
0,0001
0,8055 2: 0.0085
few rel nut
BAAI bge small 11.5
0.1250 0.0011
0.3022
0,0000
0.1350
0,0073
0,8NGO
0.00112
0.4443
0.0005
0.4455
0.000S
UJT4A ULARS
fow
ten-huae-nnssced
0.0017
0.1650
1
0.000
0.0074
0.1185
few
maseve Intere
BAAI bge small en
0,3628
0,0026
0,0685
0.000000000000 0.0000
0,6300
0,0000
0,2500
0.2079
0,096 =: 0.00MS
0.6677 0,0324
0,823 ULEUGT
masevi. Inices
sen-haas-maaeei
0.2270 0.0026
0. 10000
0.0097
0.2000
0.0X24
LOOSO
0,0095
0:108
0.0124
0.507
0,0622
0.199
0.0125
mastie_interr
unffout base
0.3244 0.0011
0.5876 0.0007
0.5130 =2 0,001
10000
0000
0,012
0A32
0.055
0.510
0.0013
masive
Mectlper-tte-nran
0.0062
0,0000
0.3762
0.0.88 0,225
lage maall-on-s1..
smull
0,4850 0.0019
0.7076 0,0007
0
0.6354 L65S0
0,7953
0/010
0.7515
maséve somati
ben- sen--hae--naeel
0.1144
0.0136
0.5621
0.0066
0,5960
0.0/24
0.5636
0,6886
0.0017
0.5585
0.1505
0.007
0.6231
intioan base
0.4412
0.0014
0.65000
0.0003
0,7000
0.0141
1.9000
-0700-0-.0000 0.7254
0.70%
0.915
0.7299 0.0000
U.714
016373
nasivv sccnutio
(ieeppe--te stallall
0.4438
0.0045
0.6765
0,0252
0,5000
0.0041
L.6011 3,4550
0,7615
= 0.0085
0,0004 BJ6T190
0.7515 2 0.6225
ntop iitent
small v1.5
0.2707
0.0000
0,6040
untioan
0.5535
0.0055 0.5131
0.10G
0.3414 1,2000-
0/0345
U.SIME 0.0U79
incai
theelpe--tee
0.2788
0.0058
0,0035
0,2150 0.0654
0,1007
0.6009 0.0185
0.6072 0,0445
0.5001
0.6137
reddit
BAAI bge small en v1.5
0.2004
0,0002
0,497 = 0,007
0,3550 0,0073
LA800 000000
0.4554 0,0015
0.5779 =2 0.0002
0,311 ULAGG
makii
ben-thuco-reneeel
0.2016
0.0084
0,1050
0,0000
0.0125
0.3061
reditt
stc-small
0,5045
0,5900
mudieschange
BAAI bgg small en 115
0,1109 :: 0,0082
0,3307 0.0111
0,1550 0,007)
0,8650 0,007]
0,4430 (,00G]
0.4449 010014
0.1745 CLEUTT
cradecrchinge
ben- -en--nn--meee
0.0029
0.0905
0.2150 000000000000000 0.0090
0,0000 2: 0.0000
0.1150 00:0000.0000 0.6211
0.3234
0.0017
0.3314
0.007
0.2774 EL 0,0015
stxliecchunee
untflogn base
0.1190
0.3405==000 0,0002
0.1000 = 0,0841
0,80G0
0.861 = 0.0005
0.4716
0.0009
0.J8IS 0.0044
mndeetslunge
0.1265
0.0088
0.1990
0.0c24
0.1669
0.0064
0.3902
Initial CAI (before HCC) and Final CAI (after HCC), along with accuracy under Zero-Shot, Single- Shot, Specialized, and HCC (R2) settings. Two clear trends emerge from the results.
First, the Final CAI (after co-alignment) is consistently higher than the Initial CAI across all datasets and all backbones, mirroring the performance improvements achieved by HCC. In par- ticular, the weakest -eade--BERT--- the largest CAI gains, with improvements substantially
27

---

Published as a conference paper at ICLR 2026
Table 16: Averaged performance over seeds {I, 42] for K 10, reported as mean + standard deviation for each Pataset-BBackoone pair.
Firat
Specisllond
benking"??
BAAL atel 1.
0.5415
0.8006
0.6687
0.00/0
0.5250
0.0496
4,5590 1. 0.2158
6.996
0.0500
0.0016 a 0.005 0.000
G.9664 &. 0.0.36
harkingt7
tem mnsde
I.1728
000340
0.2077
0.1245
0.1200
0.0455
0.5030
0.4887 BIWG
B.4536 : 0002
0.3065 &: 1-126
henking77
letflat--5--ee hone
0.0138
0:2000
0.0141
2,5000
6.00
0.701 B00E
7.785
:: 0.001
benkinu777
thetipe--pto
0.5179 0.0020
0.6007
0.0081
0.1151
0,0077
0.000
0.7734 0.0119
0.7255
clime
BAAI bgo
1:5747 0.0000
0.7065
0:0036
0.3400
0.0012
1,3000
6,881
0.6400
4 0.0044
0.5166 = BAKGOG
<lind
ber base unc
bx1350 0.0156
0,4251
0.018
0.1250
0.0071
0.083
0.032
6.015
0.00
0,6221
= 0.0011
0. = D-0010
clic:
infflaii-h---oe hoee
112430 = 0.0014
0.6601
0.0100
0:2000
0.0181
3.1000
0.0000
0.8331
0.0116
0,8083
2- 2-00001
0.3883 & (.BL
cline
thenipar-gto-
0.2002 0.0063
0,6835
0.5300
0.0081
3-523C
1
CU2213 1 0.0445
0.221
0.0002
0,7023
few nut
BAAI bge amn-hao-mncsaa
D.1201 0.0007
0.3418
0.0000
6,6224
00078
0.4407
0.005
0.3967
for yea
0.1259- 0.0304
0.3019
0.007
0.1450
0.0001
1,0890
0.0771
0. 0700
0.0046
0.881
1 0.00015
6.1177 0.9087
fon, yel yea
infleal-ch-tee
0.1265
000000
0,3722
0.0055
0.140-
0/0213
0.605
0 S281
0.0057
0.781
1 0.0055
0.0022 1. 0.0022
few Last
therlper gio small
b.1253 = 0.0000
0,3501
01005
0.0011 01350
0.0100
+ 0.025
0 6541
+ 0 0.0014
0
+ 0.0004
0.5713 100150
csassiee
D.8741
0,6181
66725
0.6881
0.6674
masiee jeiers
0.2005
0.0150
0.3197
(/112
0.150
0,040
0,3900
0.111
0.0011
0-4955
0.0000
0.3895
01077
cussive ittect
intfloat o5 base
0.2114 01030
0.506
0.014
0.2000
00811
2 6750
0 00874: 0.0078
0.2847
+ 000411
0 DU00D
cussive intect
therppes gre snall
I 0.0000
0,3896 0.0055
0.4500
02:50=0.004
0.004
200000
5 00001
0 00478
0.0078
0.0370
2 0.0007
0.EA4E = IUN04
maslie commrio
hen
1.5335-00 000000
0.5511
01131
0.5050
0,340
1.3000
0 00007
0.0117
0
4 +-0.00
0.5501
stasive scesasio
intflaat c5 base
0.6544 0,8378
0
2: 0/126
0,7080
0,001
2 0050
0.7416 0.0197
0.300
2 0.3088
0.2014 1U010
musiii joaserio
threlperggo-enal
0.6457 0.0316
0.4673
000035
0.6000
0.1151
1 0.7115
0.3361
1 1000
0710
1 0.0078
0.7973 1 0.0150
hige (og---nlll-oorl))
0.513
03:00-1
0.6305
0085
012129
000080
0,6386
0.4955
intent
inifloal 25 base
0,2040 000335
0
0.055
0.1000
1.500
0.0550
CoMS DOL48
0.6007
2 0.0245
GLE807 = 0.0125
theelpen-goo-mm ervill
019671 - 0.0176
0.000)
01002
0.300
00000
1:800
- 03131
0.5039 6 00933
0.5875
- 000111
O. 0.1719 6 D100
rocakt
BAAI BAAl-Age-andll-an--
0,2922 0.0000
0.4016 011617
03:00
1-50 - 01:275
6.226 00022
0.5712 0
0.503 BUNA4
médi
intfloate b--b
112970- 0.0111
0.409)
0.350
0,0011
0.5862
0.00015
0 010073
medi
theriper-gte small
103120-
0.0112
0,315
0.3300
0.0141
1:3130
L 00054
01031
B 0.6032
: 0.0015
0.7133 + (1000
dackenshungs
BAAl--ye-arsll-nm--
0.0039
0.0000
0.319: 0.0136
0.1100
0.0081
0-50%
0.812
0. 8397
0.0012
0-4311 1:0,0475
0.5:85
0..010
0.3077
0.330]
0004
0.2079
sackenchange
intliar- of base
1.3079
0,300
00000
0.120=
0.56%
0.001
0.0066
0.00041
dackeschuus
theriper-gto snail
0.1216 0.0002
0,3535
0.022
0.1300 0.0005
0.70% 0.6156
C.4835 0.0555
0.4647 1 0.0055
0,5796 2: 0.0.99
Table 17: Performance Comparison: HCC (R2-Best Among All Specialised Model Based on
Table 19) vs. Closed-Source Models. We compare the best HCC configuration (using the o backbone per dataset) against GPT-4o Mini and GPT-3.5 Turbo. HCC outperforms both closed-soures models on the majority of datasets. (K=3).
Dataset
HCC (R2)
4o Mini
3,5 Turbo
HCC 40 mini
HCC 3,5 Turbo
CLINC
83,98
81.44
66.58
Yes
Yes
Massive Scenario
79.49
66.83
60189
Yes
Yes
Miop Intent
75.01
64,95
Slightly
StcckEschange
47.74
51.90
30.10
No
Yes
Bunking?7
80.68
65,12
65.12
Yes
Yes
Reddit
60.62
$7,40
51.12
Yes
Yes
FewRel-NNt
47.46
35.87
32.87
Yes
Yes
Massive Intent
66.88
76.93
71.52
Table 18: Performance Comparison: HCC (R2-Best Among All Specialised Models Based on Table 20) vs. Closed-Source Models. We compare the best HCC configuration (using the opti- mal backbone per dataset) against GPT-4o Mini and GPT-3.5 Turbo. HCC outperforms both closed-source models on the majority of (K=1))
Dataset
HCC (R2)
4o Mini
3.5 Turbo
HCC 40 mini
HCC > 3.5 Turbo
CLINC
85,02
66.58
Yes
Massive Scenario
79.62
66.83
60.89
Yes
Yes
MTop Intent
66.71
75.03
64,95
No
Yes
StackExchunge
47.28
$1.90
30.10
No
Yes
Banking77
82,18
65.12
65.12
Yes
Yes
Reddit
43.43
$7.40
51.12
No
Net
47.86
35.87
32,87
Yes
Ves
Massive Intent
69.33
76.93
71.52
No
higher than those of the stronger backbones. This reflects that HCC is effective across both weak and strong model settings, and that CAI accurately captures the improvement after co-alignment.
Second, after applying HCC, the Final CAI exhibits a strong and stable correlation with the re- sulting co-aligned accuracy. This indicates that HCC not only improves the quality of the assigned annotations, but also enhances the discriminative power of CAL Consequently, CAI serves as a ro- bust, backbone-agnostee reliability signal for assessing LLM annotation quality at scale. Likewise, the Initial CAI accurately reflects baseline LLM performance across different backbone dataset combinations, demonstrating that CAT is sensitive to both model quality and task difficulty.
28

---

Published as a conference paper at ICLR 2026
Dutect
Reckleee
Irsted CAT
Fixal CAT
12M:Zeen-Sho
LLM (Sagl-Sha)
Specsliedd
BCC RCC-R2) Are.
CLANC
R35
0130
1.1
36.18
01.51
S3.11
808
4.57
27.27
80.27
$3.44
80.96
Mantre Soueado
E3
418
180
43335
53.35
13.75
13.33
TE.14
12.89
BURT
Mlap laion
(T11
11.39
1.6
35.5
23
02.50
58.34
21.04
62.47
BEXT
Socktisrhang
CTT
0:15
0.73
15:33
18.05
01.55
46.10
0.26
10.08
36.25
37.99
+3.31
BERT
18:55
0.70
17:85
31.66
38:17
71.03
OTE
0343
3.82
31.35
03.15
76.55
0.50
3155
M0.6
ВEKT
0.03
0.56
21.03
$3.50
31.24
58.13
BK2
0.33
1.5
3135
50.55
01.70
56.74
BERT
123
BowRd-Nal
E5
01.3
0.59
12333
35.31
-0.38
47.46
B0B
0.14
0.58
1333
33.99
97.28
44.54
Masire Istest
R5
0.50
3.85
1363
58.14
01.51
63:40
3.0
3430
03385
09.64
06.55
Table 19: Ablation Study across Embedding Backbones and Small Language Model (Llama8- 3 Instruct). We compare the performance HCC using three sentence encoders (E5, GTE, BGF) and small language model (BERT) across eight datasets. Initial CAI and Final CAI show how the verification signal strengthens as model quality improves. HCC (R2) consistently outperforms the Zero-Shot, Single-Shot, and Specialized baselines in most configurations, with the exception of the BGE backbone on Massive Intent and Massive Scenario. (K=3)
Datael
Backlone
Inital
Findd
Singh-Shat
Sputialiad
24.5%
CLINC
GTTE
0.59
(141
4.15
35.31
35.49
56.75
58.31
79.07
$1.91
$2.34
BGE
0.47
4.67
38.96
60.20
81.40
85.02
BEXT
0.53
1.82
44.52
48.70
59.25
75.76
Mooive Scenarto
58.94
55.04
RGB
79.03
BEXT
01.55
(199
$2.15
45.55 55
00.70
65
MTop Intent
GTE
E5
0.26
0.29
1.25
1.21
37.25
$3.58
47.04
46.55
05.02
$0:36
62 47
58
BGE
0.30
1.70
$5.59
47.61
$4.51
67.76
StackEscbange
33.13
BEXT
0.12
0.51
18.51
25.02
31.45
72.34
Banking?7
GTB
0146
0.51
2.64
2.98
33396
$1.27
51 43
03 44
71,92
70.3
72.64
76.95
77.69
Reddit
CTTE
0.36
1.71
$2.95
$1.85
$2.09
43.43
BGE
0.30
1.56
28.91
49.70
69.74
41.16
BEXT
0.12
0.73
12.61
35:18
41.16
42.92
New
11.47
32.19
0112
11.61
32:43
BEXT
0.22
0.63
23:91
12.93
$0.99
61.87
Musive Intent
GTE
E5
0.63
0.67
2.39
1.43
36,40
39.04
$7.33
53.73
$7.87
60.32
63.35
47.51
BGE
0.62
4.05
42.70
64.86
68.32
69.33
Table 20: Ablation Study across Embedding Backbones and a Small Language Model (Llama3- 8B Instruct). We compare the performance of HCC using three sentence encoders (E5,G and small language model (BERT) across eight datasets. Initial CAl and Final CAI show how the verification signal strengthens as model quality improves. HCC (R2) consistently outperforms the Zero-Shot, Single-Shot, and Specialized baselines in most configurations, with the exception of the Reddit (K=10).
In terms of effectiveness, HCC achieves the highest accuracy among all three settings, consistently outperforming the Zero-Shot, Single-Shot, and Specialized baselines. Overall, this ablation demon- strates that the CAI Ratio is both embedding-agnostic and model-agnostic: when Llama-3-88 serves as the primary annotator, HCC reliably improves both CAI and annotation accuracy across most embedding backbones. This confirms the robustness of our verification (CAI) and refinement
29

---

Published as a conference paper at ICLR 2026
(HCC) pipeline, and establishes CAI stable, reference-free free reliability signal for large-scale an- notation.
Comparison Stage (K=3)
Correlation (r)
P-value (p)
Significance
Baseline State
0.5729
6.11 x 10-4
Yes (p 0.001)
(Initial CAI vs. Zero-Shot Acc.)
Prompted State
0.6605
3.89 x 10
Yes (p < 0.001)
(Initial CAI vs. Single-Shot Acc.)
Refined State
0.7791
1.50 x 10 Yes (p < 10"?)
(Final CAI vs. HCC Final Acc.)
Table 21: Statistical Validation of CAI Effectiveness (N=32). Pearson correlation tests across all datasets and 4 backbones (3 Sentence Encoder and Language Model) show that the CAI ratio depicts reliable metric of accuracy the co-alignment process proceeds, culminating in highly significant correlation in the final co-alignment state.
Comparison Stage (K=10)
Correlation (r)
P-value (p)
Significance
Baseline State
0.5731
6.07 10
Yes 0.001)
(Initial CAI vs. Zero-Shot Acc.)
Prompted State
0.6614
3.75 x 10-5
Yes (p 0.001)
(Initial CAI vs. Single-Shot Acc.)
Refined State
0.7789
1.47 x 10 Yes (p < 10-3)
(Final CAI vs. HCC Final Acc.)
Table 22: Statistical Validation of CAI Effectiveness (N=32). Pearson correlation tests across all 8 datasets and 4 backbones show that the CAI signal becomes increasingly predictive of accuracy as co-alignment progresses, culminating in highly significant correlation the refined state.
J.4 STATISTICAL VALIDATION OF CAI EFFECTIVENESS ON ACROSS SENTENCE ENCODER AND LANGUAGE MODEL
To further validate the reliability of the CAI Ratio reference-free quality signal, conducted a Pearson correlation test between (1) Initial CAI and the LLM's Zero-Shot and Single-Shot accu- racies, and (2) Final CAI and the HCC Co-Aligned accuracy. As show in the Table 2 and Table 22 across all embedding backbones (BERT, GTE-small, E5-base, BGE-small-en-L.55 and all eight datasets, Initial CAT shows a moderate positive correlation with both Zero-Shot and Single-Shot Llama-3-8B accuracy, indicating that raw agreement between the encoders and the LLM reflects the baseline reliability of the model. After applying HCC, Final CAI exhibits a strong and con- sistently higher correlation with the resulting co-aligned accuracy, demonstrating that the HCC refinement process both increases CAI and strengthens its alignment with true correctness. These results confirm that CAI becomes more discriminative and robust predictor of LLM output quality after co-alignment, supporting its use as a backbone-agnoste reliability indicator for large- scale annotation.
J.5 RUNNING TIME AND TOKEN COST
We measured the runtime and token cost of each configuration, as summarized in Table 23 which reports the LLM running time and token usage per 10,000 examples under both zero-shot and single-shot (group prompting) settings using Llama-3-83 Instruct.
J.6 HCC RUNTIME FOR CO-ALIGNMENT
To evaluate the efficiency of HCC, as shown in Table 24 we measured the end-to-end latency of the Refinement Phase (DCCA) using Llama-3-8B Instruct. The reported runtime values are averaged across four sentence encoders (E5-base, GTE-small, BGE-small, and BERT-base).
It is important to note:
30

---

Published as a conference paper at ICLR 2026
Dutaset
Specialised Models
Llama AB Indrrcct (One-Shol)
TimeeTil raii)
Liama FAN Indractieer Shoel
Timte cain
Banking77
hunlger-glo-mmall
116.5
BAAI 3ge
Clise
BAA
fewuel.azt
base
BAAl woo.ll-g--mlllls--ooe
1.66
36.2
180
47.5
honr
musivc.inest
thenlger gre small
trafhes-ch--ee hoor
0.9
0.9
03.2
0333
0.50
102.5
96.1
BAAI age small v1.5
104
71,9
0355
1090
ben Nose unce
0.58
31.0
0:47
47.5
infilerrehh hove
0.58
34.4
0:48
08.6
thellger pic small
0.57
34.5
0.46
47.4
BAAAI bge amall v1.5
0.56
33.5
0.48
50.4
best bise uscd
1.30
01.
121.5
mispuinire
iottleor o5 hare
1.30
61.
1.37
154.2
127.9
140.3
uscased
thenlger gie
B4Al--g--ntullee--i.. hase Bg..---g-ooolll.oo -0--- -m1-115
Sunkesclongs
285.5
296.1
2.10
91.0
2.45
116.3
Table 23: Marginal Estimated Cost per 10k Examples: Comparison of Teacher Models (Llama
3 8B). M denotes millions and Min denotes minutes
These measurements exclude one-time preprocessing operations- pperations-namelyy. semantic clus- tering performed by the task-specific specialized model and general LLM annotation (both zero-shot and single-shot prompting)
are lightweight and standard.
The Refinement Phase constitutes the main active computational cost of HCC, represent- ing the core alignment process where consisteney-basee correction and re-verification oc- cur.
Table 24: Co-alignment Runtime (Llama-3-8B). We report the total runtime to process the full experimental suite sentence encoders) and the average runtime per individual encoder. The per- model cost is extremely low, averaging less than a minute.
Dataset
Total Rontime Specialised Models Encoders++Laguugge Model) Avp. Runtime per Specialised Model
Banking77
min 32.0 scc
38.0 soc
CLINC
min 15.3 scc
56.3 sec
Reddit
min 05.1
46.3
min 52.5
sec
Massive Secnario
2 min 41.9 scc
40,5 sec
Massive Intent
min 13.7
33.4
min min 55.4 sec
7.8 soc
StackExchange
min 13.9 sec
min 27
51.8
31

---

Published as a conference paper at ICLR 2026
Observation: Across nearly all datasets, single-shot prompting is substantially faster and more token-efficient than zero-shot prompting. Thus, HCC does not introduce significant additional inference cost; rather, it leverages the specialised model to perform semantic clustering and pro- vide compact in-context examples for group-based annotation, followed by a lightweight refine- ment stage. This design keeps the overall pipeline relatively efficient while improving annotation quality, At the same time, the zero-shot predictions preserve output diversity by exposing the full token-distribution behaviour of the LLM, which complements the specialised model and mitigates the overconfidence commonly observed in LLM-only predictions.
J.7 MORE DETAILS ON TWO-ROUND CO-ALIGNMENT AND CAI IMPROVEMENT (LLAMA 3-8 INSTRUCT)
Table 25 and Table reports the accuracy changes from Round 1 to Round 2 across all embedding backbones EE5-base, GTE-small, BGE-small-en-v1.55 and BERT-base-uncasele on the Llama3-8 Instruct Model. The results show that two rounds consistently provide the best balance between improvement and stability. Round 2 captures the remaining correctable cases, while additional rounds would propagate noise and degrade annotation quality.
Round 1. The initial consistent set (C) resolves the easier or more obvious annotation errors, but many harder cases in the inconsistent set (Z) remain incorrect. This is because Round relies only on C U H, so its coverage is limited.
Round 2. The samples corrected in Round expand the reference base, enabling HCC to resolve the remaining inconsistencies. This often yields further improvements (e.g., +17.7% % on Reddit with BERT).
Convergence and Signal Exhaustion (Why Stop at Round ?). Our results (Table 25 and Ta- ble 26) show that performance reaches plateau by Round At this point, the reference set has expanded to include all samples that can be reliably aligned (C U CT). The remaining inconsis- tent samples (II) are mostly "difficult" cases where the LLM and the embedding model genuinely disagree and cannot be corrected further. Importantly, going beyond Round 2 risks performance regression rather than simply yielding diminishing returns.
Across several tasks, Round already reaches performance plateau. Therefore, additional rounds tend to yield diminishing returns or even reduce accuracy.Foo example, Banking77 (BERT) shows a drop of 1.79% and StackExchange (BERT) drops by -0.56%.
These patterns indicate that the remaining inconsistencies correspond to difficult samples, making Round 2 the appropriate and stable stopping point.
Vote Weighting, Our voting procedure is already similarity aware because the top- neighbors selected using cosine similarity, ensuring that only the most semantically similar examples con- tribute to the decision. Within this top- set, however, the final vote is taken using uniform weight- ing (simple majority). More specifically, for each inconsistent instance I, we compute its similarity with every intent group.
32

---

Published as a conference paper at ICLR 2026
Dolnset (K-3)
Backbone
(CC(RI) Acc.
HCC (K2)
A
Observation
BERT
7341
71.62
-1.70
Satundion
Ranking??
ES
OTE
75422
76,85
74.58
77.08
+0.23
40.4
Sauaniiu
Comergnce
BGE
$1.35
$0.68
0.65
Satanaios
Reinemer
53.90
Stttnaton
SERT
40.41
35.13
-13.72
Comasion
Reddi
ES
GTE
43.39
43,61
60.12
60.62
+17.01
-1673
Comecian
Comesion
BGE
42.65
36.79
+14.44
Conectos
RERT
0.5819
0,6812
+00011
Relomnor
Milp Inient
E5
GTE
06190
0.4466
0.5830
0.6247
+0.1781
-00033
Regrseut
Reinmmar
0.553
0.6776
Retnemer
BERT
74.61
Sataniin
Sccnarto
78.14
77.14
06.66
+0.90
-1.00
Sauuuiio
Reinener
79.49
-1338
Stunatio
SERT
61.53
62.61
+1 08
Reiacmar
Maxdie Intent
ES
GTE
62.55
59.76
63.25
63.00
+1.21
+0.67
Reinener
Reinemar
WGB
66.47
06.30
+1 41
Retwemer
BERT
42,05
0.03
Satuntion
FowRek-Not
46.56
44.31
47.46
+0.90
Relnemen
Relnement
46.84
BERT
39.97
30.41
-0.56
Satanii
StarkFachange
46.10
47.74
+0.4
+0.25
Reinemax
Comerrsse
44.20
43.31
-0.89
Sataution
Table 25: Two-Round Correction Summary (Ilama3-8 Instruct) Across Datasets and All Spe- cialised Models. Comparing Round 1 and Round accuracy across all backbones shows that while Round provides critical corrections for complex tasks (e.g., Reddit), it represents a saturation point for simpler tasks (K=3).
Datase (k-li)
Barkhone
(CC Arc.
HCC (RZ) Arc.
A
Ubservation
BERT
0.7234
-0.0077
Resression
Banking?7
E5
0,3405
0,7098
0,7244
10.0055
..0162
Resression
BGE 04185
0,8218
Refiement
BERT
0.7058
0.0020
Consrryanee
0.8191
.00067
Reposion
0.8224
+000052
Reineneat
0.8427
0.8502
+0.0075
Reinenest
BIRT
0-0016
0.001
+008699
Reinement
Redlit
GTE
R3
0.1088
04302
0.4290
0,4343
1000641
-0000
Reinemeat
Regnosion
BGE
0,4244
0,4116
0.0108
Regression
BERT
0,5819
0,6532
+0.0713
Reinement
MTop Intent
QTE
ES
0:6211
0,6256
0.5830
0,6347
+0.0036
-00426
Reinemest
Remsosion
BCE
0:6623
0,6776
+0.0153
Reiiement
Mussve Semario
F5
0.7570
Reinement
Regnosian
038013
0,7962
BERT
0.4290
0,4202
40.0042
Reinemeal
Fre Rett
0.4638
0.4580
0,4786
0,4545
+0.0148
-0.0035
Reinemest
Repession
0.4453
0,4429
.0.0224
Repession
BEKT
06110
0.6187
Reinemest
Masnter
0,6335
0.0198
Regression
Regression
Regression
BERT
0.3591
0.3895
+0.0007
Reineneat
StackEechange
13
0.4581
0.4704
0.4625
0,4724
+00045
+010174
Refnement
Reinement
0.4189
0.4167
-0.032
Reposioon
Table 26: Two-Round Correction Summary (Llama3-8 Instruct) Across Datasets and All Spe- cialised Models (K=10).
33

---

Published as a conference paper at ICLR 2026
K ADDITIONAL ABLATION STUDIES BASED ON CHATGPT40 MINI LARGE LANGUAGE MODEL
K.1 CASE ANALYSIS OF CAI ON CHATGPT 40 MINI
Failure Case: MASSIVE-Intent. While co-alignment pipeline achieves consistent gains Banking77, CLINC, MTOP, and Reddit, MASSIVE-Intent emerges as a clear failure case, re- vealing important limitations of our approach. MASSIVE-Inten is uniquely challenging due to its dense, fine-grained intent space, where many labels differ only by minor slot variations (e.g.. inform_cityy inform_cto cit.ccttttttttteeooe info rnformmplee aeplaceecco acceedaaal.. As a result, semantically similar queries receive nearly identical embeddings, causing nearest-neighbor voting to collapse and producing unreliable correction candidates.
Furthermore, the dataset's multilingual and translationese style produces short, syntactically uni- form queries, which substantially reduces the discriminative power of English-centric embedding models. In addition, ChatGPT-4o-miní is primarily specialised for reasoning tasks and does not exhibit equally strong multilingual capabilities, further limiting its ability to exploit subtle cross-lingual intent distinctions. This is reflected in its low consistent-sample ratio -.0.4000.45) and a weak CAI signal (<1.0), indicating that disagreements between the LLM and the embed- der largely uninformative. When the consistent set is small and noisy, DCCA cannot provide meaningful guidance and instead amplifies uncertainty, resulting in limited or negative correction ef- fects. Overall, MASSIVE-Inten highlights boundary condition of method: when intent spaces are extremely fine-grained and embedding separability low, consistency-dris repair becomes unreliable. This failure case indicates future extensions, such density-adaptive clustering, slot- aware representations, and multilingual embedding backbones.
K.2 RUNNING TIME AND TOTAL TOKEN COST (CHATGPT 40-MINI)
We report the estimated token cost and runtime of ChatGPT-4o-mil across all tasks in Table 27 For each dataset and specialised encoder backbone, provide the estimated token usage per T0k examples and the corresponding estimated runtime under both single-shot (with intent) and zero- shot (without intent) prompting.
Chatse mini
Model
Singl-Shnt
Esl. Tokene (Millami
Zere-Sbul
Est. Time (Minnles)
Singl-Saot
Zere-Shat
Sctt- SEK--Bas--oa stt-aas--steeac usseaca
1,22
032
16.1
15.2
Bunking7
Seniper gic smell
auflou c5 base
0.99
1.01
0.58
0.00
15.9
16.2
17.2
17.7
BAAI oge small en v1.3
1822
0.66
16.3
16.9
Scrt- uss-acc
111
038
15.0
14.9
Cline
anflou c5 hase
Beniper giz smell
1.12
1.13
0.7
0.88
14.1
14.0
14.3
14.5
BAAI bge small en v1.5
1.11
0.98
14.3
15.0
Scrt- unsccaa
1.15
1.06
14.7
Sew rel not
intflat hase
Beniper-gte small
1.13
1.13
1,07
1.10
14.7
14.5
15.5
16.1
BAAI bge small en v1.5
1.15
1.12
14.3
16.2
Scr miaeaaa
0.76
0C75
14.7
19.
masivve inteat musive sectarão
Beniperggic-ne small
0.77
0.76
0.71
0.70
14.9
15.0
17.2
17.3
BAAl-age snall cn.51.
0.77
0.72
14.9
17.3
Sap- 4mn--ae--
0.37
12.6
11A
0.51
0.8
12.5
12.6
0.52
0.37
12.7
18.1
HAAI oge small v1.5
0.52
0.38
12.4
12.8
Scnl 2050 usceaca
107
1.
15.7
30.3
milop inneat
Beniper giz small
autlou c5 hase
1.05
1.06
1,62
1,69
16.3
16.1
29.0
30.5
BAAI bge small en v1.5
105
1.71
1577
31,0
Scrt- uuccacc
833
057
1500
13.5
Rabiit
atflea hase
Seniper giz small
0.8
0.82
0.67
0.66
14.9
14.9
15.6
15.0
BAAl bge small en v1.5
0.83
0.67
14.9
15.2
Scnt snneaac-
1nl
1.43
193
213
Stackeschange
inth aafloat-c---aas
Beniper-gte small
1.61
1.61
1.43
1.43
19.9
19.7
21.1
20.2
BAAl-age snall v1.5
1.61
1.44
20.1
20.3
Table 27: Marginal Estimated Cost per 10k Examples (GPT-4o-mini): Single-Shot (With Intent) vs Zero-Shot (Without Intent)
34

---

Published as a conference paper at ICLR 2026
K.3 RUNNING TIME OF HCC FOR CO-ALIGNMENT (CHATGPT 40-MINI)
To demonstrate the computational efficiency of Heterogeneous Consistency Co-alignment (HCC), report the actual wall-clock execution time for the complete two-round verification procedure
on all datasets in Table 28 The results show that HCC completes co-alignment within 0.36-1.11
minutes per dataset, highlighting its extremely low computational overhead.
Dataset
ChatGP DhatGPP-4o-mie I-fo-mini
Specialised Model
Time (6)
Time (min)
bert-base-uncased intilour-e--buse
33.40
0.56
37.76
0.63
Cline
thenlper-gte-saall
35.63
0,59
BAAl--gge-small-nn-11.
35.42
bert het-bbae-uncased
21.79
0.36
intlout-cS-base
28.95
0.48
massive scenario
thenlper-gte-small
0.43
1-sll-bg--smalloo--1..5
26.44
0.44
hert base base-uncased
47.42
0.79
intflout-e5-base
63.09
L,05
Stackexchange
thenlper-gte-snall
60.33
1.01
BAAI--ge-small-cm--hh
bert hase-uncased
26.25
0.44
Banking77
inflout-e5-hase
27.75
0.46
thentper-gto-small
26.54
BAAI--ge--mmallen--11
24.31
0.41
bert bas---ss-uccedd
24.72
0.41
intfloat-eS -5-hat-----se
27.49
0.46
massive_inie inntent
thenlper-gte-small
24.92
0.42
BAAl--go-small-e-v1..5
22.18
bert-base-uncised
45,91
0.77
few few_rel_nat
intfloat e5 base
66.86
1.11
thenlper-gte-small
63.53
1,06
BAAl--be--mall-on-v1..
63.22
L.05
bert- uuttbbss--ncesed
35.48
0.39
Reddit
intfoaat-e--baa
48.39
0.81
thenlper-gte steneerr-ge-maalll small
43.38
0.72
BAAll-ge-smalllenn.l...
42.57
0.71
bert-baseeuncased
31,22
0.52
intlooateS-basee
47.50
0,79
intent
thenlper gle small
44.53
0.74
MhAl-hhe-small-en-v..
41.39
0.69
Table 28: Actual Execution Time per Dataset for HCC Co-Alignment (ChatGPT 4o mini).
K.4 TWO-ROUND CO-ALIGNMENT AND CAI IMPROVEMENT (CHATGPT 40-MINI
summarises the two-round HCC correction results for ChatGPT-40r 4o-mnni across all datasets. We report (1) the initial and final CAI ratios, (2) Round-1 and Round-2 correction accu- racies, and (3) the accuracy gain from RI to R2. These results explicitly show how HCC improves annotation alignment.
Table 30
35

---

Published as a conference paper at ICLR 2026
Datasel
Backhone
Initial CAT
Final
LLMM (Zen-Shett)
1.MM (Single
Spectallod
ICC (R2)
BERT
0.70
1.36
76.82
56.85
X5.91
MGB
1.59
6.51
78.38
72.64
85.15
58.02
0.73
18.45
Masóive
GTE
1.30
64.36
73.60
79.21
BGE
1.30
3.27
61.57
09.30
76.40
73:07
0178
MTOP
GTT:
0.66
50.58
BGE
0.52
2.20
71.77
10.50
$6.57
74.54
38.57
StackEcchange
GTT:
0..00
38.98
BIDH
0.58
0.83
10.50 5M
41.33
37.15
0M:15
Banking?7
G1E
1.51
5.02
09
15.29
75.62
62.14
MGB
1.82
5.23
54.85
79.42
79:46
81. .93
Reddit
CTE
0.44
0.50
1.66
40.05
33.03
51.63
$3.34
63.34
BGB
0.47
1.58
50.30
$1.76
$1.51
60.24
FowBRekNaal
GTE
0.35
0.31
1.01
36.29
14. 64
-0.00
38.13
41.33
38.37
49.11
68
BGE
0.30
0.98
35.58
36.99
37.88
45.96
Mussève Intent
GTT
2.50
1.58
3.98
76.01
72.4
64:37
63.31
61.26
09:77
BG3
2.13
75.45
09.10
68.83
75.27
Table 29: Aggregated Backhone Comparison across 8 datasets. Initial CAI = Final CAI model's Consistent-Incoosisteet Ratio Count. Zero-Shov SSott without showing intent, Single-Shot with showing intent, Specialized = correction rate, HCC (R2) two-round majority voting accuracy (GPT-40 mini).
Dataset
Backhone
Initial
Final CAT (CC Acc.
HOC.(R2) Ace.
A Gain
Obervation
HFKT
0.20
1:34
$5.05
85.91
-002
Convergnce
CLINC
1.34
1.42
+0.34
Convergence
Convergence
BERT
0.73
1.51
78.41
77.91
-0.50
Sauuation
Masóóve Secuario
E1
CTTE
1.02
1.20
3.32
4.63
37.31
12.25
78.31
76.06
+0.50
+3.33
Satuntion
Convergence
BGI
1.10
BEXT
0.74
1.30
73.26
74.76
+1.50
Reiienaat
MTop Intent
GTE
0.65
0.68
1.59
1.70
67.85
69.81
60.50
71.45
+1.64
-0099
Retinenent
Sammatoo
BGE
0.22
2.20
73.00
74.54
+1.94
Relinemant
0.22
40.65
Cansgaanc
StackEschange
0.87
51.15
+0.21
Convegrncee
GTE
0.40
0.39
51.32
$1.49
+0.17
Sauration
RGE
0.38
0.53
49.83
40.16
-0.67
Conaggeee
Retinemeet
Bunking?7
1.51
+6.85
Connectin
Retinement
1882
5.33
-003
Conergmnce
BEKT
0.35
0.93
26.90
45.67
+13.77
Cornecton
FowRek-Nat
0.33
1,03
0.30
Convergence
BERT
0.19
0.54
42.40
59.56
+17.16
Conrecton
Rediit
GTE
E5
0.44
0.50
1.47
1,66
44.82
61.36
63.18
+16.54
+7..59
Correcton
Cenation
Conrection
REXT
0.69
0.95
30.40
70.14 14
-0.6
Summatin
Mandive Intent
GTB
1.62
1.58
2,86
2.58
69.24
69.35
89.00
69.77
0.24
0.06
Convergence
Comvergence
BGE
2.13
4.52
74.45
75.22
40.77
Refinenent
Table 30: Two-Round Correction/ / GPT-4o mini & CAT Score Summary.
L ADDITIONAL ABLATION STUDIES BASED ON CHATGPT3.5 TURBO LARGE LANGUAGE MODEL
L.I FAILURE CASE OF CAI ON CHATGPT 3.5 TURBO
No Failure Case: Our experiments show that CAI is extremely effective on ChatGPT-3.5 Turbo
compared to other LLMSs such as GPT-4o-Mini and open-source L.M Llama3-8 Instruct. CAI im-
36

---

Published as a conference paper at ICLR 2026
provements correlate reliably with downstream gains, showing that ChatGPT-3.5 aligns well with embedding-drivee cluster structure. Even extremely fine-grained settings such as MASSIVE- Intent, where cluster separability is weak, CAI correctly identifies that only marginal improve- ments are possible. The small CAI gain (0.69-0..9 aligns proportionally with the small accu- racy gain from Zero-Shot to HCC(R2) (+0.47%) demonstrating that CAT remains reliable indicator even under challenging semantic conditions.
L.2 TOTAL TOKEN USAGE AND LLM RUNNING TIME (CHATGPT-3.5 TURBO)
We report the total token usage and L.M (ChatGPT-3.5 Turbo) running time across all tasks in Table 31] For each dataset and specialised encoder backbone, we provide the estimated token con- sumption per 10k examples and the corresponding estimated runtime for single-shot (with intent) and zero-shot (without intent) prompting. These results show that HCC does not introduce signifi- cant additional token usage or latency beyond the baseline prompting scheme.
Dataws/ChutGPT 3.5 Turbo
Model
Single Shaf
Fat. lskenelikk (Millins)
Zamo-Saut
Est. Time/ik (Miunies) Stngle-She
Zume-Slan
I122
11.2
16.1
13.2
Bonting?7
thenlger-tte sio small
inflest-S5 hase
0.99
101
0.90
0.53
15.9
16.2
17.2
17.3
RAAl-ag MMMllag--
100
0896
16.5
16.9
ber hise uncesst
L11
119
13.9
14.5
Cline
Inttlex e5 base
1.12
1.13
1.97
0.95
14.1
14.00
14.3
14.3
BAAI BAAl-bye- small-----
1.11
0.95
14.3
15.0
wer-hono-mncaed
1115
1.0
fow.acl.su
thenkeer gie small
intlou c5 base
1.13
1.14
1.10
1.07
14.7
14.5
15.8
16.1
BAAI bge small en v1.5
1.15
1.12
14.3
16.2
ber hose uncased
0.28
14.7
museve.inter
Inttecr--e--ae huse
0.78
15.0
BAA Al-bg--mmallleee
intlent-
musivv sccuatio
Benbeer-gto-smi
MAAl-2g--mttllee--
intoptnnont
gie
15.7
hert-henp-mcccoe
Rasiit
14.9
BAAI
14.9
Stacleschange
Intflct- base
honbpre-tto-mmll
1.50
1.00
199
19.
BAAl--po-otilleeer11
1.90
36.1
Table 31: Marginal estimated cost per 10k examples (ChatGPT-3.5 Turbo): single-shot (with intent) vs. zero-shot (without intent).
37

---

Published as a conference paper at ICLR 2026
L.3 RUNNING TIME OF HCC FOR CO-ALIGNMENT (CHATGPT-3.5 TURBO)
We further report the wall-clock running time of HCC for co-alignment on all tasks in Table For each dataset and embedding model, list the per-dataset execution time in seconds and minutes. These measurements confirm that HCC operates within 0.36-1.15 minutes per dataset, demon- strating that the verification and correction procedure highly efficient in practice.
Dataset
Embedding Model
Time (S)
Time (min)
pert-base-uccasel
32.53
0.54
Lutfloat-e--bat
33.55
0.56
Cline
thenlper-gted
32.38
0.54
BAAI-bge-song
BAATT-be--maall-nn-11. -mall----------155
32.06
0.53
bert-base unt-bass--nneeed
22.32
0.37
intfloat-eS cS-base
29.06
0.48
Massive Scenario
sennlper-gte-maall
24.82
0.41
BAAL-bge-mmallen--11. small-en-v1.5
BAAI-bge-
24.98
0.42
brrbbas--unesedd
47.76
0.80
StackExchange
sutfoat-eS-base
58.51
0.98
thenlper-gte -tel-ssatt--lllll
53.74
0.90
BAAI- MAAl-bge-an Sel--bge--nnn small-en-v1.5
55.59
0.93
Ser-bbsc-uncased
25.88
0.43
Banking77
intfloat-e5-BB ceaat----bseee
24.55
0.41
thenlper-gte-smal
21.65
0.36
BAAI-bge-
1-Al-bbe--mall-----1..
21.61
0.36
bert-base-uncused
23.93
0.40
intfloat-e5-baa
25.66
Massive Intent
thenlper-gte-small
25.12
0.42
BAAl-bge-small-en--e
22.20
0.37
pert-base-uncused
46.35
0.77
FewRel-Nat
intfoout-e--buse
67.00
1.12
thenlper-gte-smaill
65.12
MAAl-bg--smalleen-/7
68.96
1.15
peri-base-uncused
35,28
0.59
intfoutte5-buse
44.34
0.74
Reddit
hhenlpergte-smaan
38.55
0.64
BhAl-bge-smalllen-///
39.20
uene-base-uncsed
33.70
intloutteS-buse
51.86
0.86
Mtop Intent
thenlper-gte gie- se-small
48.40
0.81
BAAl-bge-snnn aage-small-en-v1.5
45.46
0.76
Table 32: Execution Time per Dataset for HCC Co-alignment (ChatGPT-3.5 Turbo).
L.4 CO-ALIGNMENT AND CAI IMPROVEMENT (CHATGPP-335 TURBO)
Finally, Table reports the two-round HCC correction results for ChatGPT-3.5 Turbo across all datasets. We list the initial and final CAI ratios, the Round-1 (R1) and Round-2 (R2) accuracies, and the resulting accuracy change A = R2 - RI. This comparison highlights where HCC leads to refinement, convergence, saturation, regression when applied on top of ChatGPT-3.5 Turbo.
38

---

Published as a conference paper at ICLR 2026
Backhone
Initill CAI
Finul
Zero-Shit
SingleSShot
Specialized
BCC (CC(RR)
BERT
0.50
1.49
67.75
56.22
55.40
$2.76
CLENC
CIT:
E5
1.93
1.84
6.20
6.50
68.89
67:71
80.40
90.04
50.71
81:22
$6.80
$5.84
RCB
199
6.51
68.80
87.00
83:15
87.67
0.88
38.1
55.00
78.9%
Scamario
00.85
55
73.60
70
73.71
5.73
73.97
76.40
77.30
BERT
56.79
51.3
71.71
MTOP Intent
51.45
50.9%
68.40
71.71
BCE
0.67
2.11
60.37
57.73
56.57
69.51
RFRT
0.39
0.58
31:47
27.45
27-45 21-41
41 18
StackExchunge
CITE
ES.
0.77
0.74
1.25
1.26
35.68
53
38.57
38.76
38.98
18.67
48 44
47.16
BCB
0.6
1.11
34,53
37,45
37.13
45:45
BERT
74.91
Ranking??
77:48
79.45
72.63
BERT
0.24
0.59
42.56
34.19
32.05
58.
Reddit
E5
CTE
0.84
0.70
2.24
1.88
52.75
51.45
53.93
51.91
53.34
$1.63
60.71
60.58
BCB
0.83
2.19
52.10
51.79
51.51
58.19
BERT
0.35
0.55
31.14
43.46
42:50
44.91
FowRei-Nat
GIR
0.93
0.36
1.10
31.65
31.00
58.77
40.49
18.
41.25
47 12:33
30.76
37.68
45.85
Massis
71.05
61.26
68.86
68.65
BCB
2.43
4.90
69 94
69.51
66.83
72.75
Table 33: Aggregated Backhone Comparison across datasets. Initial CAI and Final CAI rep- resent
ratios. Zero-Shot accuracy without showing intent, Single-Shot accuracy with showing intent, Specialized = correction rate, HCC (R2) = two-round majority vote accuracy.
Bockhoue
ACL
Act.
Observation
0188
1.33
12:51
87.76
+0.43
Conoegerce
Cline
GTE
E5
1.84
1.93
6.30
6.29
$6.69
$5.64
36,50
85.84
40.11
40.20
Comcegetee
Comergezze
199
6.51
$2.80
$1.87
-0.32
Satundion
71.15
74.96
43.83
Masine Serronio
QTE
85
1.31
1.00
4.53
5.11
00:21
75.55
26.28
23.71
+7066
-1.54
Saturiion
Comection
BGR:
1.56
5.73
$0.95
79.30
-1.06
Satandio
BEXT
Reinemeu
STOP
Conecioa
RGF
2.11
69.00
69.81
40.41
Refazmeni
BERT
0.30
0.58
42.15
41.58
-0.58
Salandion
StackEcchunge
0.74
40.12
Comergetae
BEKT
0.55
0191
72.40
74.51
+2.41
Refmemert
Buakigg7
E5
GTB
2.03
2.54
5.15
4.88
72.05
75.88
73.96
27.44
+1.55
+1.91
Retnemert
Refacmeri
2.62
$2.11
72.63
-9.45
Ovessoot
BEXT
58.38
+17.19
Stoog Cocrection
Reddk:
GTT
ES
0.70
0.51
1.58
2.34
43.35
$3.99
60.58
60.71
+36.72
+17.22
Smog Corrctionn
Sinne Cuerzttion
BGB
0.32
2.19
42:24
58.19
+15.5
Smang Corrction
PouRai-Nor
GTE
85
0.36
0.32
1.01
47.30
48.59
47.32
40.02
Sannaiio
Conergene
BGB
0.29
0.98
16.41
45.85
-0.5
Satuation
BESCI
06.64
Comaztin
Satuttion
BGB
2.63
08.90
73.75
+3.9
Conection
Table 34: HCC Correction Summary: Initial Final CAI, R1/R2 Accuracy (in %), and Gain.
M ADDITIONAL ABLATION STUDIES ON LLAMA 5%, AND 10% USER PREFERENCE
WITH 1%,
We further analyse the robustness of HCC under varying amounts of user-preference supervision (1%, 5%, and 10%) across different random seeds using the evaluation, we additionally include two datasets, FEW NERD NAT and GO CO_EMOTION.
To broaden
We first generate annotations using the specialised model to initialise semantic clusters. Group prompting is then applied, where each query contains 10 instances belonging to the same predicted
39

---

Published as a conference paper at ICLR 2026
annotation cluster. Higher-quality ppecialised-modee annotations yield better cluster coherence, suring that semantically similar samples are grouped together before querying the LLM.
This structured grouping improves response consistency and annotation accuracy. Empirically, observe that the top 10% of LLM outputs (ranked by alignment confidence) achieve substantially higher accuracy than the bottom 1%, indicating that annotation alignment quality strongly influences downstream LLM performance.
Table 35 reports the detailed results. Across all supervision levels, HCC consistently outperforms the standalone LLM and the specialised model in most datasets. As the proportion of user user-preference supervision increases from 1% to 10%, performance improves steadily, demonstrating that HCC ef- fectively leverages limited supervision to enhance annotation reliability. Moreover, the performance variance across random seeds remains low, indicating stable behaviour under different initialisations.
Prop.
Accunccy Metrics
1%)
Specialised
Specialised
(Ours)
1%
29:26 = 1.04
54.68 = 1.24
64.85 0.13
71.18 =: 0.13
CLICC
5%
31.51 +: 1.82
65.08 1: 1.72
79,31 1: 0.47
81,36 1: 0.60
10%
36.87
1,00
70,80 1 0.02
84.38 0.40
86.81 1: 0.06
43.99 0.37
1.06
0.67
66.21 0.24
10%
46.92 2: 0.50
71.33 4: 0.63
78.53 0.22
79.22 1: 0.90
1%
28.
0.22
45.30 0.57
11.14::0.900
44.95 =: 1.79
INTENNT
5%
31.22 + 0.74
16.68 + 0.87
53.44 + 0.50
61.87 + 0.70
0.77
1.56
0.25
61.62 4.87
21.13
MIACKESURANGE
12.15 0.25
30:48 0.43
31.61
0.20
0.66
10%
14.26
1.009
34.16- 0.23
36.92
0.59
44.74 + 0.05
1%
29.23 + 1.09
50.18 + 3.39
53.73 + 3.71
60.77 1.49
1.86
1.23
19:6 0.40
12.52 1: 0.68
12.50 0.55
18,80 0.92
GRUESOTRON
3%
20.15 = 0.20
14.34 =: 0.58
14.31 +: 0.67
21.11 0.00
10%
19.68 + 0.30
18.05 0.29
17,90
0.27
26. 10 1.15
1%
11.59 1: 0.22
23.68 1.07
25.52 0.73
34.58 0.92
NEW KEL
10%
13.20 1: 0.24
0.18
36.68
0.78
1.37
1%
22.70 :: 0.47
37.17 1: 0.00
37.88 1: 0.27
$1.06 0.51
RIZCET
5%
28.31 = 0.94
49,27 =: 0.12
50.99 1: 0.26
58.80 =: 0.33
10%
31.84 1: 0.43
$2.08 1: 0,20
51,18 0.41
60.79 0.02
1.07
NEW NEHOLNA
36:11 0.07
29.66 0.24
27.62 0.01
31.31 0.12
105
35.82 :: 1.65
31.68 1: 0.93
29.89 1: 0.91
29.97 :: 0.95
156
29.06 = 1.98
44.57 =: 0.05
46.42 +: 0.59
$4.37 =: 0.33
35.13 1.37
55.83 0.76
61.72 1.06
60.15 5.35
Table 35: Lama 3-8B Instruct Performance Across Datasets with 1%, 5% and 10% User Preference Sample (Mean + Standard Deviation %)
40

---

Published as a conference paper at ICLR 2026
Dataset
Label Budget
Accuracy
(7em-Shot) (8)
Specialised
(%)
Specialised (%)
(Ours)
1%
27.28
0.06
54.39
0.06
62.86 0.94
63.36
191
Clin:
5%
34,01
2.43
68.28
1.34
78.98 0.24
82.11
0.18
10%
1.46
71.79 1.48
1.49
85.61
1%
45.15 0.49
57.30 1. 2.10
60.73 2.76
65.39
L.3
Massive Scenario
5%
46.22 0.29
66.80 1.02
75.94 +: 1,16
77.67 1.04
10%
46.36 0.86
70.10 0.57
78.50
75.03
1%
27.82 1.31
44,93 1.90
39.47 3,83
$4,00 0.63
5%
30.11 0.50
51.64 1,85
49.66 1,09
60.80 2.50
10%
32.03 0.57
57.37 0.83
55.97 + 2.30
58.40
1%
11.49 0.70
21.11 0.96
21.03 1.49
29.38 0.36
SuckEschatge
5%
12.27 0.37
29.86 0.48
31.44 1: 0.05
38.29 0.42
10%
13.16 0.28
33.20 0.67
35.73 0.38
42.82 0.42
1%
31.49 0.19
58.72 0.83
62.03 0.41
63.99 1.30
Bunking77
5%
37,08 0.06
69.71 1.43
74.17 0.776
74.32 5.29
10%
40.39 1.07
77.55 1.25
81.38 1.02
84.63 = 0.44
1%
22.58 0.22
38.25 0.89
38.67 + 1.03
50.44 = 0.20
Roldit Roddlit
5%
28,88 0.12
49.19 0.84
50.23 0.8.1
38,70 0.30
10%
29.49 1.60
52.34 0.36
53.48 0.39
61.11 0.16
1%
11.45 0.27
23.36 = 0.12
24.70
31.67 = 0.49
Roi
5%
12.46 0.25
31.46 1.22
33,69 0.84
42.40 0.28
10%
12.91 0.46
33.34 0.35
36.10 +
42.25 2.17
1%
31.61 L01
45.76 = 0.39
49.34 1.43
48.08 = 8.41
MASSIVE
5%
34,56 0.44
54.14 0.29
$8.15 0.22
61.40 0.44
10%
35.92 0.08
59.18 0.49
63.23 0.18
63.32 5.14
Lama 3-8B Instruct: Accuracy Comparison Across Datasets with 1%, 5%, and 10% Label Budgets (Mean Std, %) under Imbalanced User Preference Samples.
Table 36:
N OVERALL REFLECTION OF CAI
Interpretation. A low CAI value indicates that the Consistent subset (LLM-cncoder agreement) remains small relative to the Inconsistent subset. Importantly, this does not mean that HCC has failed; rather, CAI is correctly signalling low semantic confidence due to weak embedding sepa- ration. In such cases, the encoder struggles to form meaningful clusters, and the consistency signal becomes naturally noisier.
Implications for Practice. mend:
To enhance the robustness of CAI under low-CAI regimes, we recom-
(i) increasing the proportion of human-in-the-lowg user-preference annotations stabilise cluster boundaries, and
(ii) adopting a stronger encoder backbone (e.g., E5 or BGE) to improve representation discrim- inability.
Role as a Diagnostic Signal. Crucially, even when CAI is small (e.g., on StackExchange), HCC still produce measurable accuracy gains. In such settings, CAI serves its intended diagnostic role: it signals that improvements are likely to be moderate and that downstream predictions should be interpreted conservatively. This behaviour consistent with the broader empirical findings in Section 5.2 and Table 19 where smaller CAI deltas systematically correspond to weaker post-aligmment improvements. For clarity, we summarise practical CAI operating regimes in Ta-
ble 37
O CROSS-DOMAIN GENERALIZZATINN
Our evaluation already spans four distinct domain families, demonstrating that HCC generalizes radically different linguistic styles, noise levels, and task structures without any modification to the method. The domains included are:
Technical and Specialized:
Banking77 fne-grained financial queries (HCC (R2): 71-80%)
-StackExchange programmmig and troubleshooting (HCC (R2): 39-47%)
41

---

Published as a conference paper at ICLR 2026
Table 37: CAI Regime Guide (Empirically Grounded Interpretation). CAI measures cross- model agreement between embedding-leve structural signals and LLM token-level predictions. The regimes provide diagnostic interpretation based on observed experimental behaviour.
CAT Range
Before HCC (Diagnostic Meaning)
After HCC (Observed Behaviour)
CAI < 0.5
Very weak cross-model agreement. Embedding structure provides mini- mal support for LLM predictions. Ac- curacy is typically unstable or low. Weak-to-moderaae agreement. Some structural consistency exists, but dis- agreement still dominates.
CAI remains in this regime, im- provements are limited. Structural re- inforcement is weak gains are consistent.
CAT 1.0
Moderate CAI increases may im- prove stability, but accuracy gains are dataset-dependent
CAI 1.0 ~ 2.0
Balanced regime: agreement begins to exceed disagreement. Indicates mean- ingful structural support.
When CAI moves into or above this regime, accuracy improvements are frequently observed. especially for closed-source LLMs.
2.0 4.0
Strong agreement. Embedding simi- larity and token-level predictions are mutually reinforcing.
Typically associated with stable and substantial accuracy gains across most datasets in our experiments.
CAI > 4.0
Highh-conistency regime. Strong structural alignment between models.
Corresponds to the largest ob- served performance gains (e.g., Clinc, Banking77, Massive Scenario).
Social Media (Noisy, Informal):
Reddit subjective, noisy discussions (HCC (R2): 58 60%)
Fact-Based Structured:
Few Rew-eelNat - relation classification with structured logical semantics (HCC (R2): 42-47%)
General Assistant and Everyday Queries:
CLINC, Massive Intent, Massive Scenario, МTОР broad user-assistant intent spaces (HCC (R2): 62-83%)
Across all domains, HCC achieves moderate improvement over baselines. These consistent gains demonstrate strong cross-domain generalization.
P CAN HCC HANDLE CONFLICTING USER PREFERENCES?
Conflicting Preferences. HCC naturally accommodates conflicting user preferences because the propagation step is conditioned on each user's seed examples (H). Our method assumes that each user provides a small labeled set per class, which captures that user's individual labeling preferences. These seed examples act as the personalized anchors that guide consistency propagation and align- ment. Thus, HCC generates per-user, preference-aliee aligned annotations for large unlabeled corpora, rather than learning a global reward model shared across users.
The goal of HCC is therefore not general RLHF-style preference modeling, but Personalized Natural Language Understanding, reflecting each user's subjective preferences to unlabeled data in reliable, scalable, and wrining-free manner.
Pairwise Data. Our framework addresses categorical annotation rather than pairwise preference ranking. While CAI could, in principle, be extended to pairwise comparisons for RLHF-style data, this falls outside the scope of our cold-start classification setting. Nevertheless, we view this as a promising direction for future work.
42

---

Published as a conference paper at ICLR 2026
Q MODEL SELECTION STUDIES
Model Selection with CAT Ratio Model selection based solely on the CAI Ratio correctly iden- tifies the best-performing LLMs in 60% of cases. Among the mismatched cases, the accuracy dif- ferences are not significant. Although CAI Ratio alone is not perfect indicator of LLMs accuracy, serves as a reliable heuristic for selecting well-performing LLMs in sumi-supee settings. We have chosen the Best CAI Model and the Best Accuracy Model from the candidate LLM set, which includes GPT-3.5 Turbo, GPT-4o Mini, Google Gemini 1.5 Flash, and Llama -BB Instruct.
Best CAT Model
Bed Accuracy Model
Match
Accuracy Difference
Model
Accuracy (%)
Model
Accuracy (%)
(%)
CLINC
Google Gemini
87.24
Google Gemini
87.24
0,00
MTOP
Google
Google Gemini
75.85
StackExchange
Google Gemini
57.31
Google Gemini
57.31
V
0.00
Banking77
Gongle Gemini
73.76
GPT-3.5
73.93
X
-0.17
Massive Sccnario
Google Gemini
67.72
GPT-3.5
75.55
X
-7.83
Reddit
Google Gemini
56.23
ChatGPT-4o Mini
$7.39
X
-1.16
Emotioo
Google Gemini
ChatGPT-4o Mini
FewRel Nat
Google Gemini
52.74
Gongle Gemini
52.74
0.00
FewNERD
Google Gemini
Google Gemini
75.48
Massive Intent
Google Gemini
77.03
Google Gemini
77.03
0.00
Table 38: Model Selection Using CAI Ratio Metrie: The model selected based CAI ratio exhibits strong correlation with model achieving highest accuracy.
43

---

Published as a conference paper at ICLR 2026
R ALGORITHM TABLE
We present the algorithmic procedures for Hwide-and-Conquee Co-Aligmment (DCCA) with MV- VTES (Algorithm D and Majority Voting via the Top-Nearest Embedding Scheme (MV-VTES) (Algorithm@)
Algorithm 1: Divide-and-Conqueee Co- Alignmeett (DCCA)
Imput: Consisten setd [(7,, 1:)
Inconsistent set I (2 0/)
User-preference set H,
Number of neighbours k;
Embedding function S(-) (used in MV-VTES).
Output: Self-corrected dataset (Bnal)
Round 1: Co-aligning Z to obtain Z(1)
I())
foreach (x.g") E do
ÿ t MV-VTES(z, CUH, k, S)
Eq. equation equation
I(2) + z(2) U (()
CAI-based partition I
CT <- 0.
II -
foreach (x, 5 E do
Retrieve (I, 9) E I())
10
11
i then
CI CI ((x,0))
// Becomes consistent.
else 13
14
// Remains inconsistent
15 Round 2: Co-aligning II obtain (I())
(I()) + 0
17 foreach (x, T II do
18
I MV-VTES(r, CUCIUH, k, S)
(1 II (),
19
Final dataset
21 (final) C U CI U II())
22 return (final)
44

---

Published as a conference paper at ICLR 2026
Algorithm 2: Majority Voting via Top-Nearest Embedding Scheme (MV-VTES)
Input:
Query sample T,
Reference set D. C H & x y where (a, Des Number of neighbours k;
Embedding function S()).
Output: Refined annotation for x.
A {a E X (a. D
foreach (a, 7) E De do
S(a) S(x)
s(a, f- [S((a) (((]]
// Cosine similarity
Select {a: = TopK s(a, x) as in Eq. equation
Let Y, the label associated with a, in De
Let A (Vi,..., ) be the set of unique retrieved labels foreach E do
nh + Σ 1 = a)
// Label frequency
2 +- arg max n4a
// Eq. equation
11 return !
45

---

Published as a conference paper at ICLR 2026
S PROMPT INSTRUCTION
S.1 PROMPT INSTRUCTION (WITHOUT INTENT)
Listing 1: Prompt construction for batched intent labelling
def Prompt (prompts, specialised model labels, intention_set, temperature): unn
Build the LLM prompt for batched intent labelling. prompts: 1ist of input sentences apecialised_modell
labels from the specialised model
intention_set. candi date intent set temperature: deceding temperature for the L.M
1. Build sentence-prefixed inputs 11
combination []
12
for prompt, label in zip (prompts, specialised model__aaelll prompt1 f'For the sentence: "(prorpt)" combinaticn. (pr.icnnoppendppromttt. (prompt1)
13
15
16
2. Assemble instruction block
17
respon
respon + f"Please return exactly (len (prompts)) responses. An" respon
20
I"Identity the intention for each sentence f"using the set: Lintentionsset..We
21
22
respon +- "Follow the specified output format strictly. An" respon "Make sure the number of cutputs matches the number inputs. \a"
25
3. Query the LLM
output chpoaaal.cccccmmmlooooooeeeteete
28
model=*gptt4<-minni,, messages=[("role" "user", "content"; respon)), temperatureettemperauuue,,
29
30
max_tokens-2048,
33
34
return output
46

---

Published as a conference paper at ICLR 2026
S.2 PROMPT INSTRUCTION (WITH INTENT)
Listing 2: Prompt construction for batched intent labelling
def Prompt (prompts, specialised modee labels, intention_set. temperature):
Build the LLM prompt for batched intent labelling. prompts: list of input sentences
specialised model labells labels from the specialised model intention_sett set: candidate intent set temperature: deceding temporature for the LIM
1. Build sentence-prefieee inputs 10
I1
combination []
for prompt, 1abel in (prompps, specialised @
promptl l'For the sentence: "(prompt]" and its predicted
intent: "[specialised mrode__labels Use this predicted intent as guidance.'
14
combination. (prommmp (prompt1)
16
2. Assemble instruction block
17
respon **
18
respon += 3"Please return exactly (len (prompts) responses, \n"
respon +
f"Identify the intention for each sentence f"using the set: (intention_set)..e
21
22
23
respon += "Follow the specified output format strictly. An" respon +- "Make sure the number of outputs matches the number of inputs.
25
26
3. Query the LLM
output = model="aptt-to-mnni,, messages=[("role". "user", "content"; respon)
29
30
tomperature-temmratuue
max_tokcns-2048,
34
return output
T T-SNE VISUALIZATION FOR CLUSTERING ON ALL DATASETS FOR CHATGPT-40 MINI
In the following t-SNE visualization, the LLM-generated annotations of the identified inconsistent samples exhibit substantial deviation from their annotations. Conversely, for the identified consistent samples, the LLM-generated annotations show strong agreement with the corresponding ground-truth labels.
U USAGE OF LARGE LANGUAGE MODEL (LLM)
ChatGPT (OpenAI) was used solely for language editing improve clarity and grammar. did contribute to the scientific content, analysis, or conclusions of this work.
47

---

Published as a conference paper at ICLR 2026
SAYEP
STEPP TEL WwW.OEEEEEEE
Than www.Anateoonattee.
Figure 9: Visualization of t-SNE Clustering for LLM vs True Annota- tions on Few Lew_Ner_N Dataset. LLM outputs exhibit high similarity with ground-truth labels on consistent samples, while showing significant di- vergence on inconsistent samples.
Harenter Serate
Consinant SAYER
Consleett SHICO
Figure 10: Visualization t-SNE Clustering LLM vs True Annotations on Rei_RelNNat Dataset. LLM outputs exhibit high similarity with ground-truth la- bels on consistent samples, while showing significant divergence inconsistent samples.
48

---

Published as a conference paper at ICLR 2026
Figure 11: Visualization of t-SNE Clustering for LLM vs True Annota- tions on Massive Intent Dataset. LLM outputs exhibit high similarity with ground-truth labels on consistent samples, while showing significant di- vergence on inconsistent samples.
Incpee Semoie GLMM Acnotaationnnooooooo meces
Cluc Aondeationn 21 Pocet
ChEC Aeriialtin BY TOMES
CLI ALSAAAAAAAA AAREI
Figure 12: Visualization of t-SNE Clustering for LLM vs True Annotations on Reddit Dataset. L.M outputs exhibit high similarity with ground-truth labels consistent samples, while showing significant divergence on inconsistent samples.
49


Reviews:
[{"review_id": "9NN34fyTzd", "reviewer": "Reviewer_Q1Gc", "summary": "The paper proposes Heterogeneous-Consistency Co-Alignment (HCC), a training-free pipeline to generate preference-aligned annotations for unlabeled text by cross-checking an LLM with a lightweight, task-specific embedding model. A Consistent-and-Inconsistent (CAI) ratio—the count of agreement vs. disagreement between the two annotators—serves as a reference-free reliability signal and is claimed to correlate with accuracy. Inconsistent items are then “repaired” via a two-round divide-and-conquer self-alignment (DCSA) using majority voting over top-K nearest neighbors (MV-VTES) seeded by a small user-preference set. Experiments on eight NLU datasets with open- and closed-source LLMs report sizable gains, sometimes enabling Llama-3-8B to surpass GPT-3.5/4o after co-alignment, and the CAI ratio is empirically correlated with accuracy. (Abstract; Fig. 1 sketch on p. 2; method and metric definitions in §4; results in Tables 2–4; correlation analysis.)", "questions": "1.\tSpecialized model choice & robustness: How do results and CAI behave with different embedding backbones (e.g., E5, GTE, bge), or with weaker/stronger models? Any cross-domain generalization tests?\n\n2.\tSensitivity: Please report ablations for K, similarity metric, and cluster-seeding size; do you ever weight votes by similarity rather than uniform majority?\n\n3.\tPersonalization: Can HCC handle conflicting preferences across users for the same input? Any results on pairwise preference data (e.g., RLHF-style comparisons) rather than fixed class labels?\n\n4.\tWhen CAI fails: Provide case studies where CAI increases but task accuracy does not (e.g., StackExchange) and analyze why. What thresholds (if any) make CAI actionable?\n\n5.\tCost & latency: What is the end-to-end token/cost budget per 10k examples for zero-shot vs. single-shot prompting? How does DCSA scale with corpus size?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "•\tClear, training-free recipe that practitioners can reproduce: count cross-model agreements (CAI), then correct disagreements with k-NN style voting; the workflow is well-delineated.\n\n•\tReference-free metric: CAI is simple, label-set-agnostic, and positioned as more robust than IRR metrics in skewed/open-set settings. \n\n•\tBroad empirical sweep across eight datasets and multiple LLMs with tables showing consistent improvements (Tables 2–4) and a reported CAI–accuracy correlation. \n\n•\tLow compute barrier: avoids fine-tuning reward models and frames co-alignment as post-hoc annotation repair instead of model training.", "weaknesses": "•\tPersonalization claim is under-validated: benchmarks are mostly standard intent/topic datasets where “user preference” ≈ small labeled set per class. The paper does not convincingly show alignment to idiosyncratic user preferences.\n\n•\tValidity of CAI as a proxy: while correlations are reported, causality and failure cases are not deeply analyzed (e.g., StackExchange where CAI ↑ but accuracy ↓). A diagnostic analysis of when CAI misleads would strengthen the case. \n\n•\tDependence on the specialized model: results and CAI both hinge on the chosen sentence-embedding model. Sensitivity to the embedding backbone, domain shift, and k/neighbor choices is not fully explored.\n\n•\tReporting gaps: compute cost of group prompting, number of LLM calls, runtime, and cost/benefit vs. stronger baselines (e.g., co-training with confidence calibration) are unclear. Hyperparameter selection (top-K, neighbor weights, cluster sizes) appears ad-hoc across datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fszmYa1ELe", "reviewer": "Reviewer_VrB7", "summary": "The paper presents Heterogeneous-Consistency Co-Alignment (HCC), a training-free pipeline that lets an off-the-shelf LLM and a light-weight sentence encoder jointly annotate large unlabeled corpora while respecting a handful of user-preference exemplars. Across eight NLU datasets and three LLM families, HCC raises annotation accuracy by +2–16 %, occasionally pushing Llama-3 above GPT-3.5/4o. CAI ratio correlates strongly with true accuracy, offering a reference-free quality lever.", "questions": "1. The 'DIVIDE-AND-CONQUER REFINEMENT' is fixed at two rounds—was this simply chosen as an empirical rule of thumb? More details will be help.\n\nOther questions refer to weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The framework requires no parameter updates and only performs forward passes through two heterogeneous models so deployment is extremely straightforward.\n\n2. The CAI ratio tracks true accuracy without any labeled references and achieves remarkable Pearson correlations , indicating high statistical significance.", "weaknesses": "1. In theory, any sentence encoder (e.g., Ada-002, SimCSE, RoBERTa-base, E5) could be plugged in to replace MiniLM, but the paper provides no experimental validation of this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aFlyB12dsK", "reviewer": "Reviewer_QHKk", "summary": "This paper introduces Heterogeneous-Consistency Co-Alignment (HCC), a reference-free, training-free paradigm to improve LLM preference annotations. Specifically, HCC leverages a LLM and a task-specialized lightweight model to identify consistent and inconsistent samples in large unlabeled corpora. Moreover, the reference-free Consistent-and-Inconsistent (CAI) ratio is proposed as an uncertainty signal for evaluation, while an embedding-based majority-voting scheme enables semi-supervised co-alignment to recalibrate and propagate user preferences. \nExperiments across multiple NLU benchmarks demonstrate the effectiveness of HCC in improving annotation quality, often elevating open-source LLMs above closed-source models after co-alignment.", "questions": "My concerns are about the embedding and clustering ablations, specifically:\nCan the authors provide an ablation where the embedding model itself is weak or misaligned with user preferences? Does the CAI ratio remain effective for unreliable, out-of-domain, or adversarial specialized models? If not, how can this vulnerability be mitigated?\n\nHow sensitive are results to the choice of embedding model and hyperparameters (e.g., $k$ in majority voting, dimension reduction in t-SNE)? Will more powerful or context-tunable embeddings yield greater gains?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well motivated: this paper emphasizes the need for scalable, robust, and user-personalized annotation pipelines for LLMs that avoid reliance on costly large-scale preference annotations or ground-truth labels. \n2. The introduced Consistent-and-Inconsistent (CAI) ratio provides a reference-free, interpretable metric to quantify annotation reliability and preference alignment, addressing the annotation challenges in low-supervision or dynamically evolving settings.\n3. Extensive experimental results across eight diverse NLU datasets and multiple LLM backbones show that HCC consistently performs on par with or better than both LLM-only and specialized baselines, and even surpasses some closed-source models, which is significant.", "weaknesses": "1. Lack of evaluation on more complex NLU benchmarks such as open-ended question answering or generative preference modeling, which makes the claim that CAI generalizes as a “reference-free metric for preference alignment” may be somewhat overstated.\n2. The exclusive use of MiniLM for embedding-based clustering is only briefly justified, though results may depend heavily on this choice. The paper lacks ablations with additional embedding models or clustering variants to assess robustness.\n3. Lack of computational cost, this paper should include the time and computational costs for better efficiency analysis, does it increase the inference time?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tMEklwX87r", "reviewer": "Reviewer_MYtC", "summary": "* The authors proposed a training-free annotation method using two heterogeneous models (LLM + lightweight preference model) to verify and co-align outputs for personalized NLU tasks.\n\n* This paper introduces a reference-free metric that measures agreement/disagreement to estimate annotation reliability.\n\n* The proposed methods are validated extensively with both open and closed-source LLMs. The results show that the approach improves alignment and enables smaller models (e.g., 8B) to outperform stronger ones post-co-alignment.", "questions": "It will be really helpful if the authors can improve the presentation of the paper (especially the notations).", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "* This paper addresses an important problem in personalized LLMs. In many cases, we don't have enough trustworthy labels from users and finding ways to scale with limited labels is critical.\n\n* The evaluation is well done extensively with most of prompting baselines (Table 2-4).\n\n* The results show that the approach can make a smaller model (8B) outperform larger models (4o and 3.5 Turbo). This has important practical value and demonstrates the promise of the approach.", "weaknesses": "* The notations of the paper is very difficult to follow. The authors introduce too many different symbols throughout the paper. Many of them are not necessary (e.g. zero-shot vs. single shot). My assessment is biased by the fact that I can't follow the technical details fully.\n\n* It's unclear to me whether the approach is aligning labels or aligning LLMs. The text suggested that it's model alignment through prompt changes but when looking into the details, what the method is really doing is to use a small set of available user labels to propagate to the rest of the (larger-scale) unlabelled dataset. If this is what the authors are doing, then reward-based (e.g., RLHF) approaches are strong baselines that the paper may want to compare to.\n\n* The methods seems incremental improvements. At its core it's essentially nearest neighbor label matching and majority voting (again, due to the presentation issue, it's hard to dig deeper). It's unclear how generalizable this approach will be for real world tasks where the label space might be much larger and noisy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)