BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, and Rerankers
Authors: Ilias Aarab
Venue: ICLR 2026 Poster (2026)
Keywords: zero-shot classification
cross-encoder
embedding models
reranker
benchmark
classification tasks
NLPzero-shot classification
cross-encoder
embedding models
reranker
benchmark
classification tasks
NLP
Primary area: datasets and benchmarks
TL;DR: A comprehensive benchmark evaluating zero-shot classification performance across cross-encoders, bi-encoders, and rerankers on 25 tasks.

Abstract: Zero-shot text classification (ZSC) offers the promise of eliminating costly task-specific annotation by matching texts directly to human-readable label descriptions. While early approaches have predominantly relied on cross-encoder models fine-tuned for natural language inference (NLI), recent advances in text-embedding models, rerankers, and instruction-tuned large language models (LLMs) have challenged the dominance of NLI-based architectures. Yet, systematically comparing these diverse approaches remains difficult. Existing evaluations, such as MTEB, often incorporate labeled examples through supervised probes or fine-tuning, leaving genuine zero-shot capabilities underexplored. To address this, we introduce __BTZSC__, a comprehensive benchmark of $22$ public datasets spanning sentiment, topic, intent, and emotion classification, capturing diverse domains, class cardinalities, and document lengths. Leveraging BTZSC, we conduct a systematic comparison across four major model families, NLI cross-encoders, embedding models, rerankers and instruction-tuned LLMs, encompassing $38$ public and custom checkpoints. Our results show that: (i) modern rerankers, exemplified by _Qwen3-Reranker-8B_, set a new state-of-the-art with macro $F_1 = 0.72$; (ii) strong embedding models such as _GTE-large-en-v1.5_ substantially close the accuracy gap while offering the best trade-off between accuracy and latency; (iii) instruction-tuned LLMs at 4-12B parameters achieve competitive performance (macro $F_1$ up to $0.67$), excelling particularly on topic classification but trailing specialized rerankers; (iv) NLI cross-encoders plateau even as backbone size increases; and (v) scaling primarily benefits rerankers and LLMs over embedding models. BTZSC and accompanying evaluation code are publicly released to support fair and reproducible progress in zero-shot text understanding.

Published as a conference paper at ICLR 2026
BTZSC: A BENCHMARK FOR ZERO-SHOT TEXT
CLASSIFICATION ACROSS CROSS-ENCODERS, EM- BEDDING MODELS, RERANKERS AND LLMs
Ilias Aarab
European Central Bank Frankfurt am Main, Germany I1 flias, Aarablecb. europa. eu
ABSTRACT
Zero-shot text classification (ZSC) offers the promise of eliminating costly task- specific annotation by matching texts directly human-readable label descrip- tions. While early approaches have predominantly relied on cross-encoder models fine-tuned for natural language inference (NLI), recent advances text-embedding models, rerankers, and instruction-tunet large language mod- els (LLMs)) have challenged the dominance of NLI-based architectures. Yet, systematically comparing these diverse approaches remains difficult. Existing evaluations, such as MTEB, often incorporate labeled examples through super- vised probes fine-tuning, leaving genuine zero-shot capabilities underexplored. To address this, we introduce BTZSC, a comprehensive benchmark of 22 pub- lie datasets spanning sentiment, topic, intent, and emotion classification, cap- turin diverse domains, class cardinalities, and document lengths. Leveraging BTZSC, conduct systematic comparison across four major model families, L.LMs,
NIJI cross-encoders, embedding models, rerankers and instruction-uu encompassing 38 public and custom checkpoints. Our results show that: (i) mod- ern rerankers, exemplified by
set new state-of-the-ad with maero Fi = 0.72: (ii) strong embedding models such as GTE-large-en- en-vll- sub- stantially close the accuracy gap while offering the best trade-off between accu- racy and latency; (iii) instruction-tuned LLMs at 4-12B parameters achieve com- petitive performance (macro F1 up to 0.67), excelling particularly on topic classi- fication but trailing specialized rerankers; (iv) NLI cross-encoders plateau even as backbone size increases; and (v) scaling primarily benefits rerankers and LLMs over embedding models. BTZSC and accompanying evaluation code are publicly released support fair and reproducible progress in zero-shot text understanding
INTRODUCTION
Text classification is foundational problem in Natural Language Processing (NLP), finding broad applications across diverse domains, including topic categorization of news articles, intent detection in conversational agents, sentiment analysis of product reviews, and emotion recognition
mental health support systems (Sebastiani 2002 Kowsari al. 2019). Formally, the task involves assign- ing one or more predefined labels to textual data based solely on the content of the text (Sebastiani) 2002). However, the supervised approach to text classification necessitates the creation of large- scale, high-quality annotated datasets, process that is often prohibitively expensive, particularly in specialized domains requiring expert annotators (Settles) 2012).
"The views expressed in this paper of the author only and do not necessarily reflect those of the European
Central Bank the Farosystem.
Benchmark code and model checkpoints are available at http::://gttbb..c/m///ii
wttp://twww..cmmmmmssaaaaaaaaaaa and the live
btzsc benchmark datasets at
leaderhoard at httttss: /otttss:/huggee htg.e//ee/eooo/oooooooooo//////a

---

Published as a conference paper at ICLR 2026
Zero-shot text classification (ZSC) addresses this challenge by enabling models to predict labels that have not been explicitly observed during training (Yin et al. 2019). The core principle under- lying ZSC methods is the exploitation of semantic relationships between input texts and candidate labels. This relationship is typically captured using pretrained language models, which encode se- mantics based on extensive pretraining on large textual corpora (Yin et al. 2019 [Bronn et al., 2020). One straightforward approach involves prompting Aintruction-tuunee large autoregressive language models (LLMs) directly with textual inputs and candidate label descriptions. While effective, this method entails considerable computational cost and latency, limiting its feasibility in real-time de- ployment scenarios (Brown et al. 2020).
A widely adopted, more computationally efficient alternative involves fine-tuming pretrained encoder models on Natural Language Inference (NI.)) datasets, reframing classification tasks as entailment problems. Specifically, the input text acts premise and each candidate label as a hypothesis sen- tence (Yin et al. 2019 Bowman et al.] 2015 Williams et al. 2018). NLI datasets, including SNLI (Bowman et al. 2015) and MultiNII (Williams et al. 2018). contain sentence pains annotated with labels indicating entailment, contradiction, or neutrality. By fine-tuning encoders on these corpora, models learn to discern semantic compatibility, thus enabling effective reuse in ZSC scenarios. De- spite their success and lower computational demands relative to generative LLMs, improvements in NLI-based cross-encoder methods have plateaued in recent years.
Concurrent to this, significant advances have occurred in the domain of text-embedding models (Reimers & Gurevych, 2019; mappings, f : text + R from textual inputs to dense vector representations, ensuring semanti- cally related texts are closely situated in the embedding space. This characteristic facilitates effi- cient similarity-based retrieval, and in principle, supports ZSC through nearest-neighbor matching to candidate label embeddings (Reimers Gurevych 2019 Gao et al. 2021). The Massive Text Embedding Benchmark (MTEB) systematically evaluates embedding models across various tasks, encompassing 58 datasets categorized into eight families (Muennighoff al. 2023). However, clas- sification performance within MTEB is primarily assessed through linear probes trained on labeled data atop frozen embeddings, thereby leaving the genuine zero-shot capabilities of embedding mod- els untested (Muennighoff et al. 2023).
22199 Gao et al., 2021 Muennighoff et al. 2023). Embedding models learn
Another promising class of models, rerankers, originally cross-encoder or geuuence-to-sequene ar- chitectures designed to refine the ranking of query- document pairs (e.g., MonoT5 (Nogueira et al.) p, can similarly be adapted for ZSC by treating textual inputs as queries and label descrip- tions as retrievable documents. However, the comparative performance and potential advantages of rerankers in zero-shot classification contexts remain underexplored.
Furthermore, the distinction between mased and generative approaches is becoming increas- ingly blurred, as modern embedding models frequently leverage distilled or instruction-tuned vari- ants of generative LLMs (e.g., Sentence-TS (Ni et al. 2021), ES (Wang al. 2024h). Given these developments, a unified, controlled evaluation of all major model classes, NLI cross-encoders, em- bedding models, rerankers, and instruction-tum Existing benchmarks either rely on supervised probes (as in MTEB (Enevoldsen et al.) 22255), fo- cus exclusively on encoder architectures, or do not compare generative and non-generative methods under consistent protocol.
LLMs, under zero-shot conditions still lacking.
To address this gap, we present a comprehensive benchmark study spanning 22 datasets across four major classification categories (sentiment, topic, intent, and emotion). This benchmark systemati- cally explores the relative strengths, limitations, and transferability of these approaches, offering a comparative analysis to guide future research directions in zero-shot text classification.
2 RELATED WORK
To our knowledge, the proposed benchmark, BTZSC, is the first to jointly evaluate NLI cross- encoders, embedding models, rerankers, and instruction-tuned LLMs under a consistent, zero-shot classification protocol. Previous benchmarks for ZSC have typically been limited in scope, often restricted to evaluating single model family, narrow task category, or handful of datasets. For instance, Yin et al. (201)) introduced a foundational NLI-based ZSC benchmark but evaluated ex- clusively cross-encoder models only three datasets. Chalkidis et al. (2020) examined zero-shot
2

---

Published as a conference paper at ICLR 2026
learning specifically within multi-label classification but confined their analysis to three hierarchical datasets. Gretz et al. (2023) proposed TTC23, evaluating prompt-based methods solely for topic classification and omitted contemporary embedding and reranking models from their analysis. Lep- agnol et al. (2024) further explored the performance of smaller language models (100M-1B param- eters) across 15 datasets, yet their work excluded comparisons with embedding and reranker archi- tectures. The Massive Text Embedding Benchmark (MTEB), alongside its multilingual counterpart, has established mature, broad-ranging evaluation platform covering numerous datasets. How- ever, MTEB assesses classification performance via supervised linear probes trained atop frozen embeddings, thereby leaving unanswered the question of embedding models' genuine zero-shot ca- pability (Muennighoff et al., 2023) Enevoldsen et al., 2025 Chung et al.) (025). Consequently, this fragmented state of evaluation has hindered a clear understanding of cross-family comparative capabilities among these diverse model types.
2.1 ZERO-SHIE SHOT TEXT CLASSIFICATION
Zero-shot text classification fundamentally involves assigning labels unseen during training by sessing semantic compatibility between input texts and candidate labels, typically expressed in nat- ural language. Unlike supervised approaches, ZSC methods avoid task-specific fine-tuning by lever- aging pretrained models' semantic representations. A common parallel in vision tasks is zero-shot image recognition with language-alignn models like CLIP (Radford et al., 202T), though textual classification benefits directly from the intrinsic expressivity and flexibility of natural language doc- uments.
NLI-based cross-encoders represent one of the earliest and most prominent paradigms for zero- shot text classification. Such methods recast the classification problem into an entailment task, where each candidate label is paired with the input text as a hypothesis-premise pair scored by an NLI model (Yin et al. 2019). This approach has been operationalized effectively by public check- points like facebook htt//bartttaaa -arieemnii (Lewis et al. zero-shot pipeline of Hugging Face Transformers (Wolf et al. ing stronger encoder backbones like DeBERTa-v3 (He et al.
2020). which powers the widely used
2020). More recent advances, includ-
2023 and improved label verbaliza- tion techniques, have incrementally enhanced performance. Nonetheless, these improvements have plateaued when compared with rapid advancements from increasingly large generative language models (LLMs).
Text-embedding models have subsequently emerged as a highly active research domain, evolving significantly from early sentence embedding techniques such as InferSent (Conncau et al. (017) and Google's Universal Sentence Encoder (USE) (Cer et al. 2018). Contemporary embedding frame- works, notably E5 (Wang et al.] 2024). GTE (Li et al. 2023), BGE (Chen et al. (024)). and Qwen3- Embedding (Zhang et al.] 2025). ). have substantially raised performance standards. These models integrate sophisticated training strategies including billion-scale contrastive pretraining, multilin- gual supervision, multi-stage data scaling, and instruction fine-tuning. For example, E5 uses an instruction-tunel approach with massive-scale contrastive learning, GTE emphasizes data-scale pansion over parameter scale, and BGE combines dense, sparse, and multi-vector encoding tech- niques into multilingual framework capable of handling extensive context lengths. Compared to foundational architectures such as SBERT (Reimers & Gurevych] 2019), these advancements have resulted in improvements on standard benchmarks such MTEB, demonstrating enhanced perfor- mance in semantic representation tasks (Muennighoff et al. 2202)). Additionally, embedding models increasingly incorporate distillation from or joint-training with large generative models, effectively blurring distinctions between encoder-based and generative paradigms.
Reranker models, originally developed for information retrieval tasks, represent another promising approach for ZSC. Early reranker architectures leveraged cross-encoder models like BERT (Devlin al., 2019), DPR's combined bi-encoder and cross-encoder architecture (Karpukhin et al. 2020). and late-interaction models such as ColBERT (Khattab & Zaharia 2020). These methods typically assign relevance scores to a set of candidate documents with respect to a given input query, en- abling them to be ranked accordingly. Sequence-to-sequenn have further extended this paradigm by scoring pairs through generative token likelihood estima- tion, demonstrating effective transferability to new tasks (Nogueira et al. 2220). Recent embedding model families like BGE now provide integrated reranker checkpoints, inheriting their multi-stage training procedures (Chen et al. 2024).
reranker variants such MonoT5
3

---

Published as a conference paper at ICLR 2026
Instruction-tuned LLMs have emerged as powerful paradigm for zero-shot classification, lever- aging the general-purpos capabilities acquired through large-scale pretraining and subsequent in- struction fine-tuning. Early work by Brown et al. (2020) demonstrated that sufficiently large toregressive models could perform zero-shot classification via in-context prompting without task- specific training. Subsequent instruction-tuning methods, including FLAN (Wei et al., 2022) and InstructGPT (Ouyang et al.] 2022), further enhanced zero-shot generalization by training models to follow natural language instructions across diverse tasks. Recent open-weight models such as LLaMA (Touvron et al., 2023), Mistral (Jiang et al.) 2023), Qwen (Bai et al., 2023), and Gemma (Mesnard et 2024) have democratized access to cnerructionfolllwiin
capabilities, enabling sys- tematic evaluation across parameter scales. For zero-shot classification, these models are typically prompted with the input text and candidate labels, either selecting the label with highest generation probability or parsing a generated response (Sun et al.] (023). While instruction-tuned L.MSs offer flexibility and strong performance on diverse tasks, they typically incur higher computational costs compared to lightweight toregressive architecture, motivating research into efficient deployment strategies and smaller-scale variants (Lepagnol al. 2024).
basee alternatives due to their larger parameter counts and au-
3 BENCHMARK FOR TEXTUAL ZERO-SHOT CLASSIFICATION (BTZSC)
BTZSC presents comprehensive, aiming to serve a benchmark for diverse model architectures. BTZSC reuses subset of the dataset pool compiled by Laurer et al.] (2023) for transfer learning across a broad range of domains and task types, selecting only datasets we deem sufficiently high-quality and well-suited zero-shot
evaluation suite for zero-shot text classificatior P
text classification, and repurposes them as a standardized evaluation suite to compare diverse model families under a unified protocol. The datasets underpin five key criteria to ensure robustness and real-world relevance. First, ensuring task diversity by including at least two datasets for each of sentiment, topic, intent, and emotion classification, mirroring the four most prominent application families. Second, to probe the impact of class granularity. BTZSC covers binary, medium-sizze sized (such as agnews with four labels), and high-cardinality settings (for instance, banking77 with 77 labels). Third, we prioritized domain diversity, drawing from sources spanning news, social media, product reviews, encyclopedic content, and political discourse to assess model robustness under domain shift. Fourth, we incorporated wide spectrum of document lengths, from micro-texts (under 20 tokens) to longer articles (over 250 tokens). The benchmark is limited to English datasets; multilingual evaluation is left for future work. Full dataset details, including sources, licenses, and preprocessing, are provided in Appendix A
BTZSC comprises 22 English datasets encompassing the aforementioned task types. As summa- rized in Table [ each dataset is characterized by its number of classes, average token length and domain area (such as news, review, or social media). To quantify lexical overlap and domain simi- larity between datasets, we follow (Thakur et al., 2221)) and compute weighted Jaccard similarity by measuring token distribution overlaps for each dataset pair. The resulting 22 x 22 similarity matrix, shown in Figure I highlights low overlap between different task types, reflecting strong lexical di- versity across tasks. At the same time, we observe that datasets derived from similar sources tend to cluster more together, for example, all Wikipedia-basee datasets form distinct group, as do the biastre datasets, demonstrating modest intra-source lexical similarity.
3.1 EVALUATION METRICS
To make results comparable across all BTZSC tasks and model families, we adopt a single, task- agnostic primary metric: maero F, Macro averaging gives equal weight to every class irrespective
2 Throughouu this paper, we use the term zero-shot to denote that no fine-tuning or labeled examples from BTZSC tasks are used for training or model selection. Models are evaluated purely via document label seman- tic matching. A key limitation is that the public datasets used in BTZSC may appear in the pretraining corpora of some models; as these corpora often only partially documented, city. We mitigate this by checking publicly documented pretraining and supervised training data and avoiding models that explicitly list our datasets as supervised training targets, hut undocumented overlap may still exist. This limitation is shared with other contemporary benchmarks such as MTEB (Enevoldsen et al. 2025). comppueed with the answerdoota ModernHKKT tokenizer
cannot guarantee full corpus nov-
4

---

Published as a conference paper at ICLR 2026
Domain
Dataset
Num Classes
Avg Token Count
Emotion
dialogue
empathetic
32
132
social-media
dair_ai_emoote
6
20
Intent
banking
banking77
77
13
social-media
biasframes intent
2
27
assistant
massive_intent
59
8
Sentiment
apps
appreviews
2
49
e-commerce
amazonpolarity
2
103
finance
financialprrasebank
3
29
local-business
yelpreviews
2
164
movies
imdb
2
293
movies
rottentomatoes
2
26
Topic
education
trueteacher
2
282
news
agnews
4
politics
capsotu
21
44
politics
manifesto
56
45
qa-forum
yahootopics
10
137
social-media media
biasframes.oofensivee
2
27
social-media
biasframes sex
2
28
wikipedia
wikitoxie
2
93
wikipedia
wikitoxie_obsee
2
91
wikipedia
wikitoxie_threat
2
99
wikipedia
wikitoxic toxicaggregated
2
86
Table Summary statistics BTZSC datasets.
of its frequency, making it appropriate for both binary and multi-class datasets with varying label set cardinalities (Sokolova & Lapalme] 2209).. We additionally report (micro) accuracy, since remains the most common headline number in the classification literature and is straightforward to interpret. For more complete picture of model behavior across classes, we also report macro- averaged precision and recall for all tasks in Appendix
Finally, to probe whether success on natural- -anguage inference transfers to zero-shot classifica- tion, we evaluate each model on standard NLI benchmarks (MNLI, ANLI, WANLI, FEVERNLI, LingNI.I; details in Section 4 and report the AUROC. AUROC is threshold-free and does not re- quire calibrated probabilitics; because cosinc-similarity scores lie in [-1, 1] rather than representing probabilities, AUROC lets us test whether entailment pairs consistently receive higher similarity than neutral/contradiction pairs.
3.2 MODEL TYPES
We categorize the models evaluated in this study according their underlying architecture and training strategies.
Transformer Base Models. As a baseline, include transformer-based encoder models that have not been further fine-tuned for any specific downstream task. For these models, the final /CLS) token representation extracted and cosine similarity is used compute the relevance between the input text and each candidate label. The base models considered in this category are the original BERT (bert-large-uncavee (Devlin et al.) 2019)), the increasingly adopted ModerrBERT (ModernBERT: large (Warner et al., 2024)), and DeBERTa-v3 (deberta-v3-large (He et al. 2023)), popular and robust modification BERT that has demonstrated strong performance on a variety of NLP bench- marks.
basebased Cross-Encoders, These models are trained on NLI datasets and perform classification by assessing the degree of entailment between an input text and each candidate label, formulated as a premise- hyyohsch pair. BARR-Large-MNI included as the canonical representative. be- ing the first widely used NL.I-ased cross-encoder for zero-shot classification. We also consider

---

Published as a conference paper at ICLR 2026
1.0
Amouuipelaritty
L000.500.00
430.430
520.330
360.300
300
310.36
immb
0.541
1,00.350
160 380 310
230.
360.3630033 370. 410.000......00 060.00000000000 560 360 51
120.110..00000 5.0.50
apgeesdews we
0.430.36
400. 220.
250.21
310.3300.... 340.300
240.24
yelpeesiews
0.520 000-4000411100 000.080.0333330 222.8.55200..0 100,0 21 260.220 510.550.000.. 008.388000005577 350.370. 07000757.7700 200.27000777000 000.0/777300000000 299.00999290.
rostentooaatoss 0.330.380.24000 2.00.380.240.21..000 0 2220.2200.2033 0.060.116000 110.1......0..0 220.22200000 200.020000200 250 20t.2222.2222000 220.2220.202020. 200.220000000. 220.0220002000 222.2220.20000 230.25
dair af n'mmmtiion
0.320.310.260 $20.28
10000.00.11100 10.0016...00 0:00::0000000
220
200.27777770000 180 202.0000000000000
empatthetc
0.390.330.320.
1.0.0......0.1....... 250.190.200 200.00002222 270. 29.. 2ww.37000000000 530.33000000000
20000000000000 200.00000000000 280.086..80.22000 220.220.27
Basncialphraeehakkk 0.220.210.180. 210.2000.0000.11100
50.141.0000
130.150 180. 190.0.9...000 00001880.2000 200.190 200.230000000000 260.22
0.7
Similacity
bonkingg7 0.260.250.280.. 260,160
22200260.13
000,200 200 20.0 220
250.
240.200
150.260, 180.180..11
mastive intent 0.210.200.210 22..0..........12000 190.1199......0.
L000 1111.118..011 171.1110.000000 110.0110........000 200.200 100 000.0.........900..0 211.1.........11111
wikitoxde toxkkggroogtted 0:370.360.320 (01.:32223300200000 2wt.032.22.2.22......
200.111
www.0000300000 370.370.00000 280 20t.220..0.0.000000 24..0.22.0002
0.6
wikh wikiloxic/oksenee 0.380.370.330..0 350.2400.2000 350.2-4.210.2700000
21..61
540.000
10000 550
620.3600
370.370 000.030002000.11.330000 00..00...0000.0... 250.27
wilkitocie 'llzeat 0:410.410 1.410.410.350.660 (a..:300.000.0.22....0 20.50......90220....00000
500,300. 360.360 000.0000.0000000000 280.31
0.5
Weightel
wikitooiceinssat: 0.360.360.3111 0. 230.230.0222220 200.270. 180 200. 160 720 w0..00....
.00.3400
350.350 2ww.272...300000 200.240.266
binefrannes offeneive 0.390.970.340.3733 www.:....70..777...000.... 000.0.0000000000 200 200 360 000.300.00 360.360
310.00
630.650
260 130.033..000 130.310..00000000 240.27
blasfromes 0.390.390.330..33 370.280.270.333.....
biuddrames Tnslent: 0.390.580.340.5550 (w..00000.00222..
19 250.250.000 200 000.3700000 370.0000..00 360 (55
650
610 20000.200000000 17....0.100.240.2.
agaews ws 0.300.310.240.222 000.:2200.222.20.0 1.0...00000 w00.0.0....0 200002200 150.100.000...0
200.0280.000 200.0277.2000000 270.27
120.0440.10000 42....2.99
0.3
yahostopies 0.480.450.360 470.280.30000
3... a 2600022000 210 01..111........ 41.00.04..... 0.-.55990.430..0 22004420.40000 120.51
irueteacher 0.380.$20.20. 22..020....... 300.20000000.000000
22w.220.2....... 28
150.190.2.. 300.300.340..20
01t.330..29900 www.0..9......... 310.310
003302052
monifesto 0.310.310.40. 290.230
220.26..0.000 180. 700.24..200 2500 20w.0200.240..0
0.2
capsota 0,360, 3366.660..000 280 3 20 250.2002200 260 270, 23 21 1.090. 270 270. 310.00..00 270. 270 270 290
w00...0..0000 120.022..0 4m 100
enzzospelentty
appresiees
inch
sepprriin
meseettmmnte Fare
dais al cuuntoo mopatheese
Guandielphrr wwassnlpphmhhhatt bank
wikitodi toak- sggropote
burking?
innotive Intert
wildooale wikdorste tiktt Imsull
elaenite
biadranne elfassii
threst
biadfrana soos
blanfratnce Intert
agnniry
Inueta bar
montlent
Figure 1: Pairwise weighted Jaccard similarity between datasets.
NLI-RoBERTaabase as well as a set of tsatom-trained cross-encoders using BERT, DeBERTa-v3, and ModernBERT backbones. Both base and large versions are evaluated analyze the effect of model scale, and two loss variants are tested to assess the impact of training objectives. In total, 11 NLI-based cross-encoders are benchmarked, covering the most widely used configurations in the literature.
Embedding Models. This category comprises models optimized to produce fixed-size vector rep- resentations of text for range of downstream tasks, including classification. As canonical em- bedding model, all-MiniLM--6-12 (Reimers & Gurevych 2019) serves as a baseline for this model family. Additionally, we evaluate both base and large variants of BGE, GTE, and E5, all of which use variations of transformer encoders as backbones. To provide contrast, we also include em- bedding models that leverage large language model architectures, such as EmmeddEmng and e5-mistral-77 -b-inssttttt for Swen3--mbedddnng both O.6B and 8B parameter variants are tested to study the effect of scale. Overall, the embedding model category comprises 11 distinct models.
Rerankers. Reranker models are typically employed in information retrieval, where they re-score candidate documents for relevance to given query. The Mmmarro-MiiLL--6--1 model serves as the reranker counterpart to all-MiniLMM-6-12 and is used as the baseline for this group. Simi- larly, gee-reranker- aeeereranerrmmddenbeere modernbert-bee base and rerrrrnne--baeellarreee serve as reranking counterparts to generative reranker that
their respective embedding models. We further include Qven3-Reranke scores document-quers relevance by prompting the model to assess relevance. Both the 0.6B and are evaluated to analyze the impact of model size.
8B variants of
Instruction-tume LLMs. This category comprises autoregressive language models that have un- dergone instruction fine-tuning to follow natural language directives. For zero-shot classification, we frame the task as a multiple-choice problem: the model is prompted with the input text and a list of candidate labels, and classification is performed by computing the conditional probability of each answer token given the prompt, selecting the answer with the highest probability. We eval- uate models spanning a range of parameter scales to assess scaling behavior: Gemma-3-270m-ii

---

Published as a conference paper at ICLR 2026
and Gemma-3-Ib-s (Mesnard et al. 2024) represent the smaller end of the spectrum, Llama-3.2
3B Instruct (Grattafiori et al. 2024). Phi-4-mini--nstie instruct (Abdin et al. 2024) and Qwen3-4B (Yang et al. 2025) provide mid-scale options, while Qwen3-8B, and Mistral- Neirral--ee---aae Nemo-Instruct-22222 Instruct-2407 (Mistral 2024)) cover the larger parameter regime. We cap our evaluation at 12B parameters, as this study Tocuses on models suitable for low-latency, scalable deployment scenarios where computational efficiency remains practical constraint.
Table Appendix B summarizes the models included in the experiments, listing their architecture, training data, and parameter count. In total, the benchmark covers 38 models.
4 EXPERIMENTAL SETUP
To facilitate zero-shot classification, each class label is verbalized as a short, semantically clear, and context-rich description. For example, in the Amazon Polarity dataset, the positive class is verbalized as "The overall sentiment within the Amazon product review is {label)," where "label" is substituted with either "positive" or "negative"" depending on the ground truth.
For our custom NLI-based cross-encoders, we follow the methodology of Laurer et al. 2023 and train models on mixture of MNLI (Williams al.) 2018), ANLI (Nie al.] 2020), WANLI (Liu
iet al. 2022), FEVERNLI (Thorne et 2018)) and LingNLI (Parrish et al., 2021). datasets, deliber- ately omitting SNLI due concerns regarding data quality and label bias. Full details of the training procedure are provided in section of the Appendix. During inference we collect the entailment logits and attribute the label with the highest logit as the predicted label. For embedding models, we compute the cosine similarity between the text embedding and each label embedding, selecting the label with the highest similarity score as the predicted label For reranker models, the text to be classified serves as the query, while the verbalized label descriptions are treated candidate "documents" to be reranked according to their predicted relevance, For generative rerankers such as Qwen the scoring mechanism is detailed in Appendix C.2 For instruction-tuned LLMs, we frame zero-shot classification as a multiple-choice task, prompting the model with the input text and enumerated verbalized label options, and selecting the option with the highest next-token probability (see Appendix C.3 for details).
5 RESULTS AND ANALYSIS
In this section, we present and analyze the performance of all evaluated models on the BTZSC benchmark. Jammaiiis results across all datasets, grouped by task type, and reports (macro) F1 scores averaged within each task as well as overall, Standard deviations are included in parentheses to reflect variability across datasets. Disaggregated results with precision and recall analysis are provided in Appendix D To verify that our findings not artifacts of BTZSC's specific dataset composition, we replicate the evaluation the eight English classification tasks from MTEB v2; rankings are strongly correlated (r = 0.69, < 10 10-8) and family-wise conclusions remain consistent; for more granular details see Appendix E
addition to average (micro) aceuracy.
Base Transformer Encoders. Models that are not further fine-tuned or trained on specific semantic matching objectives perform poorly zero-shot classification tasks. Their inability to align input texts with candidate label descriptions underscores the necessity of explicit training for semantic compatibility.
Models fine-tuned on NLI data exhibit clear benefits over their off- the-shelf counterparts. Training on a diverse set of NLI datasets, including MNLI, ANLI, WANLI, FEVERNLI, and LINGNLI, yields consistently stronger performance compared to models such as bart-large-mnli and ali-roberta-baase with multi-dataset models achieving an average improvement of +6 F1 points across all tasks. Scaling model size further enhances performance: large variants outperform their base counterparts by an average +3.5 points. Tigure[la)hhgee highhigtt this differ- ence on a more granular level. Task difficulty remains dominant factor: sentiment classification is relatively easy (median Fl x 0.88-0.9), topic and intent classification are of intermediate difficulty (F1 R 0.4-0.55), and emotion detection proves most challenging (F1 ~ 0.25-0.35). Larger models
NLI-based
"For each embedding model family (E5, BGF, GTE, Fieeedimhedding). we follow the official instruction templates and queryldocument prefixes recommended in the original papers and Hugging Face model cards.
7

---

Published as a conference paper at ICLR 2026
deliver the greatest benefit for more difficult tasks, with performance gains especially pronounced in topic and intent classification. The choice of loss function, whether binary cross-entropy with neu- tral collapsed or standard three-way cross-entropy, has minimal impact overall. The only systematic deviation appears in topic classification, where the triplet variant shows degradation in performance, though intent classification continues improve under triplet training. Notably, within this family, viterta-v--large-nile
achieves the highest overall performance, surpassing both the original BERT and Mod variants, corroborating findings from Warner al. (2024) that deberta-v3 is still a challenging baseline for various NLP tasks.
does not match the performance of NLI cross-encoders (average Fl: 0.42), consistent with the historical view that NI.I fine-tuning is advantageous for zero-shot tasks. However, more recent rerankers close the gap substantially. For example, points below the best NLI cross-encoder achieves an average Fl of 0.72 and outperforms all other strongest reranker, Swee3--Rrankeet models, including NLI cross-encoders, by significant margins (+12 Fl and +14 accuracy points). This model is the top overall performer on the benchmark, ranking first in out of four task cat- egories and second in topic classification. It should be noted, however, that its size (8B parameters) far exceeds that of NLI cross-encoders (typically around 300M parameters). Importantly, even the delivers competitive results, surpassing all NLJ cross-encoders in Fl and matching or exceeding their accuracy, underscoring the strength of the reranker approach even at moderate scale.
Reranker Models. Among rerankers, the baseline Mao-marro-MiiiLMM--
achieves an average F1 of 0.58, just two and with lower variance. The
much smaller Siven--Rernnker--
Embedding Models. The canonical embedding baseline, all-MiniLM--6-12 attains an average Fl of 0.37, supporting prior observations that rerankers generally outperform embedding models in re- trieval, albeit at higher computational cost. However, newer embedding models such as e5 ea-rarge-v2, achieve substantially higher F1 scores (0.60, 0.59, and
gte-modernbert- base, and gte-large-en-a 0.62, respectively), placing them on par with or even surpassing the best NI.I cross-encoders. No- tably, these embedding models lack Pross-attention between documents and label verbalizers yet still deliver strong results at similar model sizes. For instance, gte-large-en-Vi encoders and all rerankers of comparable size, yet it still trails the top-performing 8B by roughly 10 F1 points. Scaling up embedding models does not yield the same improvements observed in rerankers; for example, (ieen--Embedding -8B only marginally improves over its 0.6B variant (F1: 0.59 vs. 0.58).
surpasses all NLI cross- -rren-Reranner-
Instruction-tuned LLMs. Instruction-tunee LLMss form fourth family, evaluated in a strictly zero- shot setting with simple prompt templates. Their performance spans a wide range and reveals that both scale and model family play critical roles. Small models such as gemma-3-22 3-270mit and gemma- 3-1b-it perform comparably to base encoders (average Fl 0.28 and 0.36, respectively), indicating that sub-billion parameter LLMs struggle as zero-shot classifiers. At the 3-4B parameter range, and 1ii-4-mini-instree achieve only mid-0.40s F1, roughly matching weaker NLI cross-encoders and embedding baselines, whereas Qwen3-4B reaches 0.65 F1 despite comparable size. This performance gap suggests that at moderate scales, Instruction-tuning quality and base model design matter more than parameter count alone. At 8B+ parameters, L.MMs become consistently competitive: Qwen3-8B achieves 0.66 Fl with accuracy around 0.71. The strongest LLM, Mistral-Nemm Veemo-Instruct- Fl and 0.71 accuracy, surpassing all embedding models and NLI cross-encoders, though it remains about 5 Fl points behind the specialized Owen3-Rerankers LLMs particularly excel on topic classification (F1 up 0.69), while lagging somewhat intent and emotion relative to the best rerankers and embedding models.
we observe substantial variation by model family: 3.ama-3.2233--mat
(12B), attains 0.67
Figure| b)) further elucidates scaling trends. All three families benefit from scaling, but with dis- tinct regimes. Rerankers exhibit roughly monotonic gains with scale and form the top curve at all sizes, culminating in Owenn--Rernnkerr million parameters and then largely saturate around 0.60-0.62 F1. LLMs show the steepest scaling: performance rises slowly at sub-billion scales and then sharply between 3B and 8B, where they catch up with the best embeddings and approach the strongest reranker. Figure 1 plots model F1 score against normalized inference speed (inverse wall time) on a standard test set. The upper-right quadrant, bounded by the medians of both metrics, highlights models that best balance accuracy and efficiency. The majority of the models in this region are embedding models, indicating they offer the most favorable trade-off between performance and latency for practical deployments, with
Embedding models improve rapidly up to a few hundred
8

---

We also examine whether NLI task performance predicts zero-shot classification effectiveness. As shown in Figure [b the relationship is strongly model-family dependent. For NLJ-tuned cross- encoders, there is a clear, almost linear relationship: higher NI.I AUROC consistently translates into higher Fl on BTZSC, reflecting the direct transfer of entailment supervision to zero-shot classifi- cation. Large LLMS follow a similar pattern, with models that perform better on NLI also achiev- ing stronger BTZSC results, indicating that antailment-aligneee reasoning capabilities in instruction- tuned LLMs remain predictive of ZSC quality. Rerankers, although not explicitly fine-tuned on NLI, still display a positive trend: better NLI AUROC is generally associated with higher BTZSC F1. At the same time, several rerankers attain strong classification performance despite only moderate NLI scores, suggesting that dlevance-focused training captures discriminative task signals that standard NLI benchmarks do not fully reflect. Embedding models, by contrast, show tightly clustered NLI AUROC values but wide spread in BTZSC FI. This lack of a clear monotonic relationship implies that once a basic level of NLI competence is reached, NLI performance is no longer a good proxy for zero-shot classification quality in embedding models. Instead, the structure of the embedding topical distinctions are hypothesized to be the dominant Published as a conference paper at ICLR 2026
gte werranker-mdddrnbeerr base as the only reranker achieving comparable efficiency. Large LLMs, in contrast, tend to be accurate but slow: they cluster in the upper-left region of the plot, well outside the Parcto-efficient quadrant.
1.0
Task
sopic
rnthesting trdder
sentineent
LL3G
0.8
50000
141
0.4
0.4
0.2
base
large
large (triplet)
panumeters (ailkias log scale
(a) NLI-based cross-encoders performance.
(b) Scaling across model sizes.
Figure 2: (a) Performance of NLI-based cross-encoders on BTZSC; points are individual datasets and diamonds mark task-wise medians, comparing model size (base vs. large) and loss type (binary vs. three-way). (b) Effect of scale on zero-shot performance: macro-Fi F1 vs. parameter count (log scale); bands show 95% Cls.
5.1 NLI PERFORMANCE, AS A PROXY FOR ZERO-SHOT CLASSIFICATION
space and its ability to encode factors.
Madd
all croes macodee
Ternekee
eecholdinng Model
nil cosss eczodey
hodbiiding moddl
0.7
0.7
0.6
HIZSS sccuze
0.6
BTZSU
0.5
0.4
0.3
2.00
0.10
6.06
0.66
0.10
0.10
0.5
will
time
NLI score
(a) Accuracy vs. latency trade-off.
(b) Relationship between NLI ability and ZSC.
Figure 3: (a) serformance-speed trade-off: macro-F; on BTZSC vs. normalized inference through- put w/wall time) on standard test set; the upper-right quadrant (split by metric medians) marks models with the best accuracy-effiienccy balance. (b) NLI ability vs. zero-shot classification: AU- ROC standard NLI benchmarks (x-axis) BTZSC macro-F, (y-axis).

---

Published as a conference paper at ICLR 2026
Model
Topic
Sentiment
Intent
Emotion
Avg Fl
Avg Acc
Base encoders
bert-large-uncasee
0.34 (0.22)
0.38 (0.06)
0.15 (0.24)
0.08 (0.11)
0.30 (0.20)
0.40 (0.26)
Jeberta-y3-large
0.30 (0.23)
0.34 (0.03)
0.16 (0.26)
0.05 (0.07)
0.27 (0.20)
0.36 (0.25)
ModemBERT-lange
0.30 (0.24)
0.37 (0.06)
0.14 (0.21)
0.03 (0.04)
0.27 (0.21)
0.35 (0.24)
NII cross-encoders
bart-largo-mnli
0.37 (0,23)
0.84 (0.19)
0.43 (0.16)
0.41 (0.04)
0.51 (0.28)
0.53 (0.28)
ali-roberta-base
(0.25)
0.80 (0.15)
0.30 (0.22)
0.33 (0.02)
0.49 (0.28)
0.50 (0.28)
sert-haseunneasd--nt
0.43 (0.27)
0.76 (0.17)
0.30 (0.28)
0.26 (0.15)
0.49 (0.29)
0.50 (0.28)
hert-lagg-uuncuscd-lii
(0.27)
0.79 (0.10)
0.35 (0.26)
0.27 (0.21)
0.53 (0.28)
0.57 (0.28)
pert-larg-unncase--ll---i-
0.49 (0.27)
(0.12)
0.35 (0.27)
0.24 (0.06)
0.52 (0.28)
0.55 (0.26)
Sebberta-3-basc--ne
0.49 (0.26)
0.86 (0.10)
0.31 (0.17)
0,33
0.55 (0.28)
0.58 (0.26)
deberta-v3-lan -arage---
0.48 (0.26)
0,90 (0.06)
0.48 (0.17)
0.44 (0.00)
0.59 (0.27)
0.62 (0.25)
vechert--3-large-nlaae
0.50 (0.28)
(0.07)
0.45 (0.23)
(0000)
0.60 (0.28)
0.62 (0.27)
modembeert-ase-nll
(0.26)
(0.44)
0.27 (0.18)
0.29 (0.01)
(0.29)
0..690
moodembertt-laree-tt
0.47 (0.25)
0.86 (0.16)
0,40 (0.21)
0.30 (0.00)
0.55 (0.28)
0.59 (0.27)
moodenbeett-ange--l---iii
0.45 (0.26)
0.88 (0.12)
0.41 (0.
0.34 (0.04)
0.55 (0.29)
0.58 (0.27)
Rerankers
ms tee-marco-MMinn -iniL
0.41 (0.14)
0.59 (0.16)
0.30 (0.29)
0.19 (0.01)
0.42 (0.20)
0.46 (0.21)
geraanke-moddrebee--beee
0.49 (0.14)
0.82 (0.17)
0.51 (0.15)
0.42 (0.04)
0.58 (0.20)
0.62 (0.19)
ngerreranker-hase
0.42 (0.14)
0.62 (0.15)
0.47 (0.04)
0.29 (0.00)
0.47 (0)
0.49 (0.14)
age-reranker-larg
0.43 (0.19)
0.78 (0.15)
0.54 (0.07)
0.37 (0.03)
0.53 (0.22)
0.56 (0.21)
ween-RReranker-O66
0.54 (0.24)
0.80 (0.20)
0.55 (0.07)
0.45 (0.06)
0.61 (0.23)
0.64 (0.21)
Owen Rennnnkee----8
0.66 (0.19)
0.92 (0.06)
0.70 (0.03)
0.49 (0.00)
0.72 (0.19)
0.76 (0.15)
Embedding models
all-MimiLLM-L6-v2
0.41 (0.12)
0.35 (0.04)
0.41 (0.07)
0.13 (0.03)
0.37 (0.12)
0.44 (0.14)
e5-base-v2
(0.20)
(0.19)
0.56 (0.08)
0,40 (0.04)
(0.23)
0.62 (0.21)
e5-large-v2
0.50 (0.17)
0.86
0.55 (0.04)
0.41 (0.04)
0.60 (0.22)
0.62 (0.20)
c5-mistral-77-insttee
0.41 (0.21)
0.87 (0.13)
0.65 (0.02)
0.50 (0.00)
0.58 (0.26)
0.62 (0.24)
bge-base-on-vl.5
0.47 (0.20)
(0.20)
0.58 (0.05)
0.36 (0.09)
0.57 (0.24)
0.59 (0.22)
bge-turge-en--1.5
(0.19)
(0.19)
0.58 (0.09)
0.39 (0.06)
0.55 (0.25)
0.59 (0.24)
gie-base-en-vl.5
0.49 (0.23)
(0.18)
0.59 (0.07)
0.37 (0.07)
0.58 (0.24)
0.61 (0.23)
gte-large-cn-v1.5
0.54 (0.22)
0.85 (0.18)
0.59 (0.04)
0.37 (0.04)
0.62 (0.23)
0.64 (0.22)
gte-modemmbert-bse
0.45 (0.20)
0.87 (0.12)
0.62 (0.01)
0.42 (0.04)
0.59 (0.24)
0.61 (0.23)
Owen3-FinbeediingD..0
0.49 (0.14)
0.81 (0.17)
0.56 (0.10)
0.43 (0.07)
0.58 (0.20)
0.61 (0.18)
Owen3-Embedding-BB
0.44 (0.16)
0.89 (0.09)
0.59 (0.
0.51 (0.05)
0.59 (0.23)
0.64 (0.20)
Instruction-tumed LLMs
(0.21)
0.42 (0.11)
0.13 (0.22)
0.04 (0.04)
0.28 (0.21)
0.31 (0.21)
gemma-3-Ih-it
0.34 (0)
0.52 (0.14)
0.24 (0.25)
0.14 (0.02)
0.36 (0.20)
0.40 (0.21)
L.amm-3,2-3B-Imm Instuutntt
0.44 (0.16)
0.46 (0.09)
0.41 (0.05)
0.35 (0.03)
0.43 (0.12)
0.46 (0.11)
Qwen3-4B
0.64 (0.23)
(0.11)
0.40
0.37 (0.08)
(0.25)
0.70 (0.22)
Phi-4-mini-instrud
0.44 (0.15)
0.49 (0.13)
0.37 (0.07)
(0.01)
0.43 (0.14)
0.47 (0.14)
Qwen3-8B
0.65 (0.23)
0.90 (0.08)
0.48 (0.
0.32 (0.11)
0.66 (0.25)
0.71 (0.23)
Mistral-Nemo- Nww.o-omstttttt-4-0000
0.69 (0.24)
0.84 (0.17)
0.46 (0.17)
0.36 (0.10)
0.67 (0.25)
0.71 (0.23)
Table 2: Zero-shot classification results on BTZSC. We report macro Fl per task family, overall macro F1 (Avg F1), and micro accuracy (Avg Acc). Parentheses report the across-dataset standard deviation within each task family reflecting task heterogeneity. Bold indicates the best and under- lining the second-best score per column; the best model within each family (based on overall macro FI) is also underlined.
6 CONCLUSION AND FUTURE WORK
This paper introduces BTZSC, a unified benchmark for zero-shot text classification that jointly eval- uates NLI cross-encoders, embedding models, rerankers, and instruction- -uned LLMs. Across 22 datasets, modern rerankers achieve the best overall performance, while strong embedding models of- fer the most attractive accuracy-latency trade-off: NLI cross-encoders remain competitive but show diminishing returns with scale, and NLI scores predict zero-shot quality only within this family and for LLMs, not for embeddings. Instruction-tuner LLMs at 4-12B parameters form a fourth regime: clearly better than base encoders and small LLMs and competitive with strong embeddings and cross pross-encoders. yet still lagging behind the best rerankers at substantially higher cost. BTZSC, together with released code and models, provides a reproducible testbed for future work on mul- tilingual extensions, improved label verbalizations and prompts, and scaling up instruction-tun -tunedd and reranker models for realistic zero-shot deployment.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY STATEMENT
To foster transparency and enable future work, we release the complete evaluation benchmark, including preprocessing and evaluation scripts, with the full codebase under an MIT license at The BTZSC datasets are available at btttpps: trained model checkpoints at https:// gittrub.ccom/Tlaaaakaaabbbbba and live leaderboard at hhtpps: pistps://hugeer http:///gggggga////e/ee/ All training configurations (optimizer, learning rate schedules, batch sizes, and carly stopping criteria) documented in the appendix. For robust- ness, training results are averaged over three independent runs with different random sceds. Eval- uation was performed execution across models. All inference runs were carried out bfloat precision, and training, where applicable, employed mixed precision for efficiency. Pretrained checkpoints sourced from public libraries are properly cited and referenced. To our knowledge, no restrictions prevent full reproducibility of the results presented in this work.
co/ h//p///css/bbasss//tts-eeeeeeeee
compute cluster with NVIDIA A100 80GB GPUs, using parallelized
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, et al. Phi-4 technical report. arXiv preprint urXiv:2412.08903 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint htt.:2239916609, 2023.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher Manning. A large anno- tated corpus for learning natural language inference. In Proc. EMNLP, pp. 632 022-622,44 2015.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, and el al. Language models are few-shot learners. arXiv preprini arXiv:2005. 141:: 2020.
Iñigo Casanueva, Tadas Temeinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders. In Proceedings of the 28th International Conference on Computational Linguistics, 2020.
Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, et al. Universal sentence encoder. In Proceedings of EMNLP, 2018.
Ilias Chalkidis, Manos Fergadiotis, Sotiris Kotitsas, Prodromos Malakasiotis, Nikolaos Aletras, and Ion Androutsopoulos. An empirical study on large-scale multi-label text classification including few and zero-shot labels. In Proceedings of EMNLP, 2020.
Jianly Chen, Shitao Xiao, Peitian Zhang, Kun Luo, Defu Lian, and Zheng Liu. Bge m3-embedding: Multi- Multi-lingual, multi- multi-functionality. multi- qulti-granularity text embeddings through self- oelf-knowledge dis- tillation. arXiv preprint arXiv:2402.032. 2024.
Isaac Chung. Imene Kerboua, Marton Kardos, Roman Solomatin, and Kenneth Enevoldsen. Main- taining mteb: Towards long term usability and reproducibility of embedding benchmarks. arXiv preprint arXiv:2506.22 1882, 2025.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. In Proceed- ings of EMNLP, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, 2019.
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, Márton Kardos, Ashwin Mathur, et al. Mmteb: Massive multilingual text embedding benchmark. arXiv preprint arXiv:2502.13555 2025.
Jack FitzGerald et al. Massive: A Im-example multilingual nlu dataset with 51 typologically-diverse languages. In EMNLP, 2022.
11

---

Published as a conference paper at ICLR 2026
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Proc. EMNLP, pp. 091046691,2000 2021.
Ziv Gekhman et al. Trueteacher: Learning factual consistency evaluation with large language mod- els. In EMNLP, 2023.
Giovanni Grano, Andrea Di Sorbo, Francesco Mercaldo, Corrado A. Visaggio, Gerardo Canfora, and Sebastiano Panichella. Android apps and user feedback: A dataset for software evolution and quality improvement. In Proceedings of the 2nd ACM SIGSOFT International Workshop on App doi: 10. 10.1145/1121264.322222
Market Analytics ( (WAMA), pp. 8-11, 2017.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
Shai Gretz, Alon Halfon, Πya Shnayderman. Orith Toledo- Ronen, Artem Spector, Lena Dankin, Yannis Katsis, Ofir Arviv, Yoav Katz, Noam Slonim, and Liat Ein-Dor. Zero-shot topical text classification with Ilms - an experimental study. In Findings of EMNLP, pp. 9647-9676, 2023.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentanglee embedding sharing. In Proc. Interna- tional Conference on Learning Representations (ICLR), 2023. httptttttss:ppeeeeeeeeee
hutt/oorum?id-333 - arXiv:2111.09544
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.068255 2023.
Bryan D. Jones, Frank R. Baumgartner, Sean M. Theriault, Derek A. Epp, Rebecca Eissler, Cheyenne Lee, and Miranda E. Sullivan. Policy agendas project: State of the union speeches. https://wwas wwwp:oorraaiiieeeennaaaa.. net/ 2023. Comparative Agendas Project dataset.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering. In Pro- ceedings of EMNLP, 2020.
M. G. Kendall and J. D. Gibbons. Rank Correlation Methods. Oxford University Press, New York,
NY, 5 edition, 1990.
Omar Khattab and Matei Zaharia. Colbert:
Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of SIGIR, 2020.
Kamran Kowsari, Kiana Jafari Meimandi, Mojtaba Heidarysafa, Sanjana Mendu, Laura E. Barnes, and Donald E. Brown. Text classification algorithms: A survey. Information, 10(4):150, 2019. doi: 10.3390/info 10040150.
Moritz Laurer, Wouter van Atteveldt, Andreu Casas, and Kasper Welbers. Building Efficient Uni- versal Classifiers with Natural Language Inference, December 2023, URL http::/arxiv:
org/abs/ 2312. 17543 arXiv:2312.17543 [cs].
Pola Lehmann, Simon Franzmann, al. The manifesto data collection, version 2024a. In WZB Data Release, 2024.
Pierre Lepagnol, Thomas Gerald, Sahar Ghannay, Christophe Servan, and Sophie Rosset. Small language models are good too: empirical study of zero-shot classification. In Proceedings of CREC-COLING 2024.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeee pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL, 2020.
Zehan Li. Xin Zhang, Yanzhao Zhang, Dingkum Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.0328. 2023.
12

---

Published as a conference paper at ICLR 2026
Alisa Liu, Swabha Swayamdipta, Noah A. Smith, and Yejin Choi. WANLI: Worker and Al collab- oration for natural language inference dataset creation. In Findings of EMNLP, pp. 7080-7097, 2022.
lyy Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Represenlations, 2019. URL
http://wwwwwwwweeeeeeeeeeeeeeeee forum? ?id=
Bkg6RiCqY7
Andrew L. Maas, Raymond E. Daly, Peter Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In ACL, 2011.
Pekka Malo et al. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of Emerging Technologies in Accounting, 2014.
Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint 2rXi::2403300029: 2024.
Mistral AI. Mistral nemo, 2024. cessed: 2024-07-18.
-nemo Ac-
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB; Massive text embed- ding benchmark. In Proc. EACL, pp. 2014-2037 2023.
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. Sentence- Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.0885 2021.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial NLI: A new benchmark for natural language understanding. In Proceedings of ACL, pp. 4885-4901, 2020.
Rodrigo Nogueira, Zhiying Jiang, and Jimmy Lin. Document ranking with a pretrained sequence- to-sequence model. arXiv preprint arXiv:2003.06713, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhangg Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. ACI., 2005.
Alicia Parrish, William Huang, Omar Agha, Soo-Hwan Lee, Nikita Nangia, Alex Warstadt, Kar- manya Aggarwal, Emily Allaway, Tal Linzen, and Samuel R. Bowman. Does putting a linguist in the loop improve NLU data collection? In Findings of EMNLP, pp. 4886-4901, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceed- ings of the 38th International Conference on Machine Learning (ICML). PMLR, July 2021.
Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. Towards empathetic open- domain conversation models: A new benchmark and dataset. In ACL, 2019.
Nils Reimers and Iryna Gurevych. Gentence-BERG Sentence embeddings using siamese BERT- networks. In Proc. EMNLP. 2019.
Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. Social bias frames: Reasoning about social and power implications of language. In ACL, 2020.
Elvis Saravia, Hsien-Chi Toby Liu, Yan-Hao Huang, Junlin Wu, and Yi-Shin Chen. Carer: Con- textualized affect representations for emotion recognition. In Proceedings of EMNLP 2018, pp- 3687-3697, Brussels, Belgium, 2018. doi: 10. 10D1865/
13

---

Published as a conference paper at ICLR 2026
Fabrizio Sebastiani. Machine learning in automated text categorization. ACM Computing Surveys, 34(1):1-47, 2002.
Burr Settles. Active learning literature survey, of
Technical Report 1648, University
2012. URL
activelearning.g
Marina Sokolova and Guy Lapalme. A systematic analysis of performance measures for classifica- tion tasks. Information Processing & Management, 45(4):427-437, 2009.
Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang, Text classification via large language models. pp. 8990-9005, 2023.
Nandan Thakur, Nils Reimers, Andreas Rückléé Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models, 2021. URL
https: org/ als///as//////// 2 04 08663
James Thorne, Andreas Vlachos, Christos Christodoulopoulls, and Arpit Mittal. FEVER: A large- scale dataset for fact extraction and VERification. In Proceedings of NAACL-HILT, pp. 809-819,
2018.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2:X1v:230..1397,,
Liang Wang, Nan Yang, Xiaolong Huang. Linjun Yang. Rangan Majumder, and Furu Wei. Multi- lingual E5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672, 2024.
Benjamin Warner, Antoine Chaffin, Benjamin Clavié, Orion Weller, Oskar Hallström, Said Taghadouini, Alexis Gallagher, Raja Biswas, Faisal Ladhak, Tom Aarsen, Nathan Cooper, Griffin Adams, Jeremy Howard, and Iacopo Poli. Smarter, better, faster, longer: A modern bidirec- tional encoder for fast, memory efficient, and long context finetuning and inference, 2024. URL https: //rxivv orgg abs 2412.411222 13663
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc Le. Finetuned language models are zero-shot learners. Interna- tional Conference on Learning Representations, 2022.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proc. NAACL-HLT, pp. 1112-1122, 2018.
Thomas Wolf, Lysandre Debut, Vietor Sanh, Julien Chaumond, Clément Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, oct 2020. Associ- ation for Computational Linguistics. doi: 10.18653/v 10t033/v1120022 wttp:/..2.0mpn//... wetp:0000.mmmpmmmmo..bb... URL https: //aclaathhooogy.org/2222
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. Ex machina: Personal attacks seen at scale. In Www, 2017.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Wang, Bowen Zheng, et al. Qwen3 technical report. arXiv preprint ar05v:2505. 200.22080000088 2025.
Wenpeng Yin, Jamaal Hay, and Dan Roth. Benchmarking zero-shot text classification: Datasets,
evaluation and entailment approach. In Proc. EMNLP-IJCNLP, pp. 3914-3923, 2019,
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In NIPS, 2015a.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In NIPS/Zhang et al. (015a)). Yahoo Answers Topics subset.
14

---

Published as a conference paper at ICLR 2026
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In NIPS Zhang et al. (2015a). Yelp Reviews Polarity subset.
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, al. Qwen3 embed- ding: Advancing text embedding and reranking through foundation models. arXiv:2506.00116. 2025.
arXiv preprint
A DATASETS OVERVIEW
BTZSC comprises 22 English single-label classification datasets spanning topic, sentiment, intent, and emotion, as described in Section and Table I Each dataset is treated as separate, single- label task: we never merge examples or label spaces across datasets, and all metrics are computed per dataset before aggregation.
All datasets are publicly available through Hugging Face Datasets and listed in Table 3 together with their original sources and licenses. BTZSC reuses subset of the dataset pool compiled by Laurer et al. (2023) for transfer learning across a broad range of domains and task types, select- ing only datasets we deem sufficiently high-quality and well-suited to zero-shot classification. Where applicable, we adopt their task definitions and label verbalizers.
A.1 INSTANCE FORMAT AND SPLITS
For every dataset D, we standardise the Hugging Face representation to simple pair
(x4, v:) E X x {o...., Lp 1},
where is single input text string and is categorical label index.
Single text field. Each original dataset may expose one more textual fields (e.g. text, sentence, utterance, comment.text). We map these to single canonical text field as follows:
If there is single obvious document field (e.g. text, review, comment.text), use that field directly as zę.
If relevant information is split across multiple short fields (e.g. title + body, question + answer, conversational context), we concatenate them in fixed order with newline separators to form one string zi. No dataset-specific prompts or instructions are injected.
This standardisation is performed once per dataset and then reused across all model families, so every model sees exactly the same input text for given example.
Single categorical label. Each example carries a single gold label. We map the dataset-specific an integer index ys
label field (e.g. label, sent iment, topic, intent. emotion) {O...., Lp 1}. where Lp is the number of classes in dataset D. The mapping between origi- nal label names and indices fixed per dataset and shared across all models. Multi-label datasets are not included in BTZSC.
Splits and zero-shot protocol. For each dataset we evaluate purely zero-shot on the designated evaluation split. Whenever the Hugging Face dataset exposes an official test split, we use that split as BTZSC's test set. For datasets without separate test split, we adopt the dev or val I i dat i on split as test. No remixing or re- p-rriittiniing of examples is performed.
No labeled examples from any BTZSC dataset are used for training, hyper-parameter tuning, or model selection. All reported scores are computed on these test splits under the zero-shot protocol defined in the main text: models are only allowed to use their pretrained parameters and generic evaluation prompts/verbalizees, with no supervision from BTZSC labels.
https:///uggingffccccc////aa
15

---

Published as a conference paper at ICLR 2026
A.2 LABEL VERBALIZERS
sero-shot classification is implemented via natural-language label descriptions. Following Laurer ct al. (203)), each dataset is associated with:
set of class names (e.g. positive, negative, neutral, toxic, non-toxic), and short, semantically informative verbalizer template describing how the label appears in context.
Concretely, for each dataset we define a template such
"The overall sentiment within the Amazon product review is {label)"
where (label} is replaced by the class (e.g. positive negative). The exact templates and class names are inherited from the release of Laurer et al. (2023).
These verbalizers are shared across all model families:
For NLI cross-encoders,, the text acts as premise and each verbalized label as hypothesis.
For embedding models, both x; and verbalized labels are encoded and compared via cosine similarity.
For rerankers, 2; the query and verbalized labels are candidate "documents".
For LLMs, verbalized labels appear options in multiple-choice prompt (Appendix C.3
Crucially, we do not tune verbalizers per model or per dataset: the same set of label descriptions is used for all runs ensure strict comparability.
A.3 TOKENISATION TRUNCATION
All models use their own official tokenizer. For each batch, we determine maximum sequence length
Thatch = min [Tmmodel, max [
where Emodel is the maximum context length supported by the model and | is the tokenised length of input x; in that batch.
Inputs longer than T.model truncated the model's hard limit; otherwise manual truncation applied. We use dynamic padding to the longest sequence in the batch and do not apply any additional dataset-specifie preprocessing.
A.4 EVALUATION AND AGGREGATION PROTOCOL
Evaluation proceeds in two stages.
Per-dataset evaluation. For every dataset D we compute:
macro-averaged Fj over the Lp classes,
micro-averaged accuracy,
macro-averaged recall, and
macro-averaged precision
on the full test split of D. No labels or examples are shared across datasets, and there is no aggrega- tion of label spaces.
16

---

Published as a conference paper at ICLR 2026
Aggregation across datasets. To obtain the summary scores reported in Table only ag- gregate dataset-level metrics. For each task family compute the family-wise macro-F, the unweighted mean of macro-F, over all datasets D F. The same unweighted averaging is used for fame accuracy. Each dataset therefore contributes equally, independent of its size class cardinality. We never pool examples across datasets when computing these aggregates: there is no global micro-averaging over all test instances. Instead, BTZSC deliberately treats each dataset an independent testbed and uses simple unweighted aver- ages to summarise performance across this collection.
(topic, sentiment, intent, emotion]: J. we
A.5
SOURCES AND LICENSES
Table 3 lists the original sources and licenses for all datasets used in BTZSC.
Domain
Dataset
Source
License
Emotion
dialogue
smpathetic_dialogues dialogues dair ai emotion
Rashkin et al.
2019
CC BY-NC 4.0 Rescarch/edncation only
social-media
Saravia et al.
2018)
Intent
banking
banking77
Casanueva et al.
2020
BY 4.0
social-media
biasframes intent mattin
Sap el al. 2020
CC BY 4.0
assistant
FitzGerald et
2022
CC BY 4.0
Sentiment
apps
appreviews
Grano et al.
2017
Unknown
e-commerce
amazonpolarity
Zhang al
2015a
Apache-2.0 CC BY-NC-SA 3.0
finance
financiaphrraebbank
Malo el al.
2014
local-business
yelpreviews
Zhang et
2015
ToU (non-commercial)
movies
imdb
Maas
2011
IMDb Non-Commercial Terms
movies
rottentomatoes
Pang Lec
2005
CCO
Topic
education
trueteacher
Gekhman et al.
2023
CC BY-NC 4.0 Non-commercial
news
agnews
Zhang ct
2015a)
politics
capsotu
Jones et al.
2023
Laurer et al.
2023
CC BY-NC-SA 4.0
politics
manifesto
.chmann
al.
2024
Terms Use
ga-forum
yahootopics
Zhang el al.
20136
Unknown
social-media
Hhasframes.offensie
Sap el al.
2020
CC BY 4.0
social-media
biasframes
Sap al.
2020
CC BY 4.0
wikipedia
wikitoxic iinsutt
Wulczyn ct al.
2017
CCO 1.0
wikipedia
wikitoxic_obscaar
Wulczyn ct al.
2017
CCO 1.0
wikipedia
wikitoxic .threat
Wulczyn et al.
201
CCO 1.0
wikipedia
wikitoxic liaeagggeet
Wulczyn et al.
2017
CCO 1.0
Table Sources and licenses BTZSC datasets. All datasets are used as single-label classification tasks and evaluated in purely zero-shot setting their respective test splits (Section A.1
A.5.1 DATASET DESCRIPTIONS
Below provide brief descriptions of each dataset included in BTZSC, outlining the source do- main, annotation scheme, and classification task.
dialogue empathetic. EmpatheticDDalogue is corpus of multi-turn conversations in which one speaker describes personal situation and the other responds in an empathetic way. Each dialogue turn is associated with one of several emotion categories (e.g., joy, sadness, fear), so the dataset can be used for emotion topic-style classification of conversational text.
dair-ai adir.ai.eootio emotion. The dair.ai.ende inn dataset corresponds to the dair-aa emot in corpus, collection of short English texts (originally social media posts) annotated with discrete emotion labels. Each instance is labeled with one of six basic emotions (anger. fear, joy, love, sad- ness, or surprise), making it standard benchmark for single-label emotion classification in English.
17

---

Published as a conference paper at ICLR 2026
banking77. Banking77 is dataset of short customer queries to online banking assistant (e.g.. "I want to freeze my card"), each labeled with one of 77 fine ine-grained intent classes such as card issues, transfers, account information. It is designed for intent classification in the banking and financial customer-support domain.
biasframes intent. These splits are derived from the Social Bias Frames dataset, which contains English sentences about social groups (often from online platforms) annotated with rich labels describing the speaker's intent and implied meaning (e.g.. whether they intend to offend, to joke, to express hatred, etc.). The intent view focuses on predicting those communicative-inten labels from the text alone.
appreviews. The app_revi ews dataset consists of user reviews from mobile app stores (e.g., Google Play / Apple App Store), where each review is paired with sentiment-style label that reflects the user's overall evaluation of the app (for example negative vs. positive, or star-based ratings). It is used as product-review sentiment classification benchmark.
amazonpolarity. Amazon Polarity is large-scale sentiment dataset built from Amazon prod- reviews. Each example is review text labeled as either positive or negative, based on the original star rating, and spans a wide range of product categories (books, electronics, etc.).
financialphhasseaan. Financial PhraseBank contains short English snippets from financial news and company press releases, each labeled according to the sentiment of the text with respect to the target company's future performance (positive, neutral, or negative). It is a standard benchmark for fine-grained sentiment analysis in finance.
yelpreviews. The Yelp review datasets are collections of user-written reviews of local busi- nesses (restaurants, shops, services) posted on Yelp. Depending on the variant, each review is la- beled either with binary polarity (positive vs. negative) or with one of several star-based rating categories, making a benchmark for review sentiment classification.
imdb. The IMDB sentiment dataset consists of movie reviews from the Internet Movie Database, each labeled as positive or negative based on the overall opinion expressed. Reviews are relatively long and varied style, so the dataset often used to test document-level sentiment classification.
The Rotten Tomatoes movie review dataset contains short snippets of film reviews taken from the Rotten Tomatoes website, each annotated with binary positive/negative sentiment label (in some variants, a finer-grained rating). classic benchmark for sentence- level sentiment classification.
cottentomatoes.
massive. MASSIVE ("Multilingual Amazon SLU Simulation for Slot filling, Intent classifi- cation, and Virtual assistant Evaluation") dataset of crowdsourced virtual-assistant utterances spanning many languages. Each utterance is labeled with an intent class and slot annotations; in this benchmark is used for intent classification on assistant-style queries.
trueteacher. The TRUETEACHER dataset is large collection of sentence pairs constructed to study and improve factual consistency in summarization. It is built from news summarization corpora where a powerful language model (FLAN-PaLM 540B) is used to generate candidate sum- maries and then label them as factually consistent or inconsistent with their source articles. The resulting pairs support training and evaluating models that distinguish faithful summaries from hal- lucinated unsupported content.
agnews. AG News is topic-classification dataset of news headlines and short descriptions col- lected from the AG's News corpus. Each article is categorized into one of four high-level topics: World, Sports, Business, Sci/Tech.
18

---

Published as a conference paper at ICLR 2026
capsotu. The CAPSOTU dataset is derived from the Comparative Agendas Project's State of the Union (SOTU) data o The SOTU corpus breaks U.S. presidential State of the Union speeches into short quasi-sentences and codes each segment with detailed policy content categories.
manifesto. The Manifesto dataset is derived from political party election manifestos compiled by the Comparative Manifesto Project. Text segments (often sentences or quasi- sentenesst are an- notated with policy-topic categories such as economic policy, welfare, or foreign relations, enabling multi-class classification of political positions.
yahootopics. The Yahoo Topics (Yahoo! Answers) dataset contains questions and accompany- ing text from the Yahoo! Answers platform, each assigned to one of several broad topical categories (e.g., Society & Culture, Science & Mathematics, Sports, Business & Finance). It is used multi-class topic classification benchmark for user-generated Q&A text.
biasframes offensive. This split of the Social Bias Frames data focuses on labels describ- ing whether an utterance is offensive or not, and to what degree. The underlying texts are short statements about social groups, annotated for perceived offensiveness, so the task classify lan- guage according to its offensive content.
biasframes sex. This variant the Social Bias Frames dataset isolates labels related to sex- ism baseer-based bias. The texts again statements about people groups, annotated for whether they convey sexist stereotypes or implications, turning the task into detecting gender-related bias in language.
wikitoxic insult. These splits are based on the Wikipedia Talk Labels toxicity datasets re- leased in the Jigsaw toxicity challenges. The insult subset contains comments from Wikipedia talk pages, each annotated for whether it includes insulting language, and is used as binary classi- fication task for the presence of insults.
wikitoxic obscene. The obscene split from the same Wikipedia toxicity data contains comments annotated for obscene or vulgar language. The classification task is to determine whether given user comment uses obscene expressions not.
The threat split consists of Wikipedia talk-page page comments labeled ac- cording whether they contain threats of violence or other threatening language. used train and evaluate models on detecting threatening content in online discussions.
Aakitoxic.threaa
wikitoxic.toxi ttx.ccgggreaateddd The toxicaggregated variant combines several toxicity-related labels (e.g., toxic, toxic, obscene, insult, threat, identity hate) from the Wikipedia toxicity datasets into single binary "toxic vs. non-toxic" label. This yields a broader notion of toxicity for classifying harmful comments in online conversations.
B MODELS OVERVIEW
Table 4 provides comprehensive overview of all 38 models evaluated in this study, including their architecture, backbone, fine-tuning data, parameter count, and pooling strategy.
19

---

Published as a conference paper at ICLR 2026
Model
Yi
Arch.
Backbone
FT / train data
AP
Pool /iim
Base encoders
bett lanee-u---de
2018
BERT
NA
340M
2021
enc.
DeBERTa
NA
304M
ModlerBBERT-laree
2024
ModernBERT
NA
395M
NLI cross-encoeers
burt- large-maii
2020
enc-dec.
BART
SNLL, MINLI
406M
nli-roberta hase
2020
RoBERTs
SNLI MNLI
125M
bert-baseunnnnelle
enc.
BERT
MNLI, ANLI WANLI FEVERNIL, LINGNLI
110M
bert-large -n-t-arg--mnnod--li
enc.
BERT
above
bett- sange-unased-ali--nee
enc.
BERT
same as above
340M
deberta-y2. vorrt-----buse
EnC:
DeBERTa
Abberta-3-laage-
DeBERTa
above
301M
deberta--3-large ali-triplet
enc.
DeBERTa
sume as above
304M
modernbert-buse. huse-nli
ModernBERT
above
149M
modernberi--arge-ee
cnc.
ModernBHR1
same as alooe
395M
modernbert--ngg-all--te
enc.
ModemBERT
Rerankers
ms mns-maroo-io Minil M-L.6-v2
2021
enc.
Minil M
MS MARCO
22.7M
glo-reranker modeeke--oddeeee--bbbeeee
one.
ModernBERR
large multiingg pairs
bge-reranker-base
2023
enc.
XLM-RoBERTa base
large maltiling, pairs
278M
bge reranker-large
enc.
RoBERTa large
large multiling, pains
Qwen3 Reranker-O.66
2025
dec.
Qwen
syntheric yev'no ranking
0.6
Qwen's Keranker-885
dec.
Qwen
syntheric yeshno ranking
Embedding modelss
all MisiLL--66-62
2021
enc.
MiniLM
IB puized sentences syntheric contrustive
22.7M
mean / 384
e5-buse-v2
enc.
(BERT)
mean
c5-large-v2
2023
(HERT)
same above
335M
mean/ 1024
---mistral-7--instroot
2024
dec.
Mistral-78
synthctic multiling contrastive 1.5B pair data, contrustive same as above
7B
Jast 4096
bge e..
2023
BGE (RoB.)
137M CLS/ 768
bge-large-en-vl..
2023
enc.
BGE (RoB.)
434M
CLS 1024
gle gt--buse-en-v155
en2.+
contrastive pretrain same above
gte-large-en-v1.5
2024
enc.+
GTE
434M
CLS / 1024
ge-modernbert-huce
enc.
ModernBFRT
766
Qwen3-Ean Qwen3-Ennsediin Embbedding--e
2025
dec.
Qwen3
synlhetic multiling, contrastive synthetic mulilling. contrastive
0.68
Jast/ 1024
Senns--Emeediing-3 SB
2025
dec.
Qwea3
SB
Jast 4096
LLMs
3-200a
2025
dec.
Gemma 3
NA
270M
3c--ma-3-1b-it
2025
dec.
Geama 3
NA
1B
Llama-3.2 3.2-3B-Instrus
2024 dec.
Llama 3.2
NA
3,2
Qwen3-4B
2025
dec.
Qwen3
NA
4.0B
Phi-4-mini-insrract
2025
dec.
Phi-4
NA
3.88
Qwen3-8B
2025
dec.
Qweaš
NA
8.28
Misrral- Nemo--ommm-e22200
dec.
Mistral-Nemo
Table 4: Architectural and training overview of the 38 models evaluated. Columns list publication year (Yr), encoder/decoder architecture (Arch.), backbone, principal fine-tuning data, parameter count (#P), and pooling strategy with embedding dimensionality.
C EXPERIMENTAL SETUP
C.1 TRAINING PROCEDURE FOR NLI CROSS-ENCODDNN ENCODERS
Let a paired input sequence (premise || hypothesis) be tokenised as x = (zu (CLS) .i...., [SEP)....,.a and encoded by pretrained Transformer backbone So : NS RSX with hidden size E:
H fo(x) SXE
h Hy E RR (CLS row).
A two-layer classification head with dropout p 0.1 transforms h:
h Dropout (h),
u W,, 2 LayerNorm(u)). L = W22 + b2,
WIEREXE, b ERR
33)
ExC R2
(4)
where C is the number of label logits.
20

---

Published as a conference paper at ICLR 2026
Binary variant. Here C and R is entailment logit. The probability of entailment is (1) (1 + + and the model is optimised with binary cross-entropy:
LBCE(U: E) y o
v) log(1 o(f), y E {0.
Three-way variant (tripler). Now C' = 3 with logits (Lents Eocuts fconta). During training the standard multi-class cross-entropy used:
exp(fy)
CyE(y, L) log > exp(lx)
y E [1,2,3}
During evaluation the scalar entailment score is
Lent log(E fue + e eLumm))
which is the log-odds of ENTAILMENT versus the union of the other classes (with probability (s)).
Validation signal. Early stopping is triggered by the dev-set loss computed on an equal-sized,
balanced union
Daev = MNLI M AN U A U ANLI, U WANLI FEVERNLIU
At every evaluation step the loss is measured, and training stops when this loss fails to decrease for 10 consecutive evaluations, or 3 epochs, whichever comes first. Evaluation is performed every 1% of total steps.
Optimiser and schedules. Fine-tuning uses the PyTorch AdamN (Loshchilov & Hutter 2019 optimiser with default settings (31 0.9, B₂ 0.999, 10-s weight-decay 0.01). The
learning rate employs a linear warm-up for the first 10% steps followed by cosine decay:
0.17
T(L - ((1-0.17)
2 1/0
cos
0.17 SIST.
0.97
with separate initial rates for the backhone (Mene) and classification head (Thea))-
Large backbones:
Tene 8 x 10 Thead 4 x 10
Base backbones: TJenc
10-5, Mhead 10-4
All models train for E epochs with mini-batch size B 32 and no layer freezing.
C.2
QWEN3 RERANKER
For every query-document pair we build single decoder-only prompt of the form
P P=prefix: + (Instruct): / + (Query): q + (Document): d + suffix.
Fixed strings.
Prefix
ilimsstart/ >system
Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be "yes" or "no". <lim_end|> <lim_start >user
Suffix
<lim_end|>
< <lim_start >assistant
<think>
</think>
21

---

Published as a conference paper at ICLR 2026
Instructions I.
NLI retrieval
Given a piece of text, retrieve the passage that entails the text the best.
Label retrieval
Given piece of text, retrieve relevant label descriptions that best match the text.
C.2.1 BINARY DECISION VIA "YES/NO" TOKENS
Let Tyes and Tho be the token IDs that realise the strings "yes" and "no". Denote the final-step logit vector by v Ls-1 R.V We extract
Tyos Urw
and compute the entailment probability
Pyes
eryys + e Fran
C.3 INSTRUCTION-TUNED LLMS
For zero-shot classification with instruction-tumer LLMs. we frame the task as multiple-choice problem. Each candidate label is assigned a unique single-token character option (e.g., "A", "B", "C", etc.). For every input text and label form:
(. y.)), we construct prompt of the
P systtm.iinstruutioo + (Tex)): + (Options): O suffix
where O enumerates each label verbalizer with its assigned character option.
Fixed strings.
System instructions
You are text classifier. You will be given a text and several mutually exclusive options. Each option is prefixed by single letter (e.g. A, b, ...). Your task is to choose the single best option.
IMPORTANT:
Answer with EXACTLY ONE LETTER used to prefix the options. Do NOT output any words, punctuation, or explanation.
Text and options format
TEXT: (text) }
OPTIONS:
A) (label_verbalizer__1
B) [[abee_verbalizee__21
Suffix
Answer: The correct option is letter
22

---

Published as a conference paper at ICLR 2026
C.3.1 CLASSIFICATION VIA NEXT-TOKER PROBABILITIES
Let (ai.....au axk be the set of single-token character options corresponding to the K candidate labels. We tokenize each symbol aę to obtain its vocabulary index Tax.
Given the constructed prompt P. we perform a single forward pass through the decoder-only model to obtain the logit vector at the final position:
Ls E IR
where V is the vocabulary size and S is the sequence length.
We extract the logits corresponding to the option characters and compute a softmax over this re- stricted set:
exp(Urar)
pluk | x))
Σラ= exp(Ura3)
The predicted label is then:
Ú arg max ke k).
p(yk
This approach requires only a single forward pass per unique input text (rather than one pass per text-label pair), making it substantially more efficient than per-label scoring methods while still leveraging the model's 's instruction -iirructioo-folooiing capabilities.
D DISAGGREGATED RESULTS
To better understand where different model families succeed fail, we complement the main results with disaggregated scores by dataset and metric. Tables S report macro-averaged Fi, micro- averaged accuracy, and macro-averaged recall and precision for each BTZSC dataset and model.
Overall, the disaggregated view confirms the aggregate picture from the main benchmark. Base encoders perform poorly and inconsistently across datasets, with low and systematically weak recall and precision, especially on intent and emotion tasks. NLI-tuned cross cross-encoders and rerankers form the strongest families: they attain uniformly high F,, recall, and precision on most topic and sentiment tasks, and maintain relatively strong performance on more challenging intent datasets. Embedding models and instruction-tune
LLMS sit in between: the best embeddings (e.g. GTE-, BGE- e5-, and Qwen3-bass models) and stronger LLMs (Qwen3 and Nemo) reach competitive Fi scores, but the disaggregated metrics reveal systematic variation across domains.
For the embedding models, the hierarchy already visible in aggregate metrics becomes particularly clear. The baseline all-MiniL.M-L6-v underperforms with average recall around 0.45 and precision around 0.50, and struggles notably high-cardinality nuanced datasets such BANKING77, BIASFRAMES- -NNTEETTE--INEETT, EMOTIONDIARY. and EMPATHETICDIALOGGUS models les-base/large, BGE-base, GTE-base, Owen33-Emhedding-0.66 achieve average recall and precision the low-to-mid 0.60s and deliver very strong performance on classical sentiment tasks (Amazon Polarity, IMDB, RottenTomatoes Yelp, FinancialPhraseBaak)) Top-tier embeddings such as gte-large-en-vl. and Qwen2-Embeddiing-BB further improve both recall and precision (Avg R/P 0.67-0.70) and show more stable behavior across topic, sentiment, and intent tasks. Their per- formance on emotion and still outperforms weaker embeddings and base encoders.
In contrast, mid-tier
intent remains clearly below that on standard sentiment, but
The disaggregated scores also highlight dataset-level effects. Standard sentiment benchmarks (Ama- zon Polarity, IMDB, Rotten Tomatoes, Yelp, FinancialPhraseBBank) are close to saturated for all strong models: NLI cross- cross-encoders, rerankers, top embeddings, and strong L.MMs all reach recall and precision around 0.9 or higher, indicating that these tasks are no longer particularly discriminative for modern architectures. In contrast, several BTZSC datasets expose sharp differences. The Mani- festo dataset is consistently hard across families, with markedly lower recall and precision even for the strongest models, reflecting the difficulty of multi-class political text classification. BiasFrames (offensive/sex//ieen)) and WokiToxic variants likewise reveal non-trivial gaps: while top rerankers and embeddings achieve good performance, smaller weaker models often show pronounced
23

---

Published as a conference paper at ICLR 2026
asymmetries (e.g. reasonable recall but poor precision, or vice versa), especially BIASFRAMES- INTENT. For intent classification, BANKING77 and MASSIVE again favour rerankers and strong embeddings, with instructionttund LLMs performing competitively but not surpassing them. Fi- nally, the emotion datasets (EMOTIONDIARY. LMPATHETTCDLLLOOUUR
are the most challenging slice for all families, with recall and precision typically 0. 0.1-0.2 points lower than sentiment and topic tasks, even for the strongest models.
Comparing recall and precision directly, we do not observe a systematic family-wide recall- precision trade-off.. Top embeddings, rerankers, and LLMs tend to be reasonably balanced, with mild tendency for the best embeddings and LLMs exhibit slightly higher precision than recall on the hardest tasks. Rerankers, in particular, are recall-preserving intent datasets, while maintaining acceptable precision, which desirable for high-recall applica- tions such as safety filtering. Taken together, these disaggregated results show that BTZSC can be used not only to rank models by single headline metric, but also to characterise their error profiles across domains (news, political text, product reviews, social media, dialogue) and task types (topic, sentiment, intent, emotion). In this sense, BTZSC provides a nuanced testbed for analysing how different model families trade off recall and precision across diverse classification landscape.
toxicity and social-
24

---

Task
Topic
Sentiment FPB
Intent
Emotion
Published as a conference paper at ICLR 2026 2026
Model
AGN BE-OIT BE-Sex CAPS
MAN
TT
WT-Ins WT-Obs
WT:Thr WT-Agg
YT
AmzPol
AppR
IMDB
RT
Yelp
197
BR-Int MASS
Emol)
Emp
Avg F1
Base encuders
bert bert-large uncased
0.17
0.48
0.48
0.03
0.01
0.33
0.51
0.58
0.49
0.53
0.08
0.42
0.36
0.28
0.38
0.47
0.36
0.02
0.42
0.00
0.16
0.00
0.30
deberta-v3-large
0.18
0.49
0,12
0.01
0.01
0,34
0,55
0.56
0,51
0.53
0.03
0.33
0.37
0.30
0.33
0.35
0.37
0.01
0.45
0.00
0.10
0.00
0.27
ModemBBERT-larg
0.10
0.53
0,46
0.01
0.01
0.36
0.46
0.58
0.13
0.60
0.08
0.33
0.44
0.30
0.34
0.36
0.43
0.02
0.38
0.02
0.06
0.01
0.27
NLI cross-encoders
hart- hart-large large-mnii mnli
0.71
0.37
0.07
0.33
0.09
0.51
0.33
0.69
0.08
0.55
0.27
0.93
0.92
0.47
0.93
0.83
0.96
0.28
0.60
0.41
0.44
0.38
0.51
ali-robert--bse
0.69
0.47
0,18
0.14
0.02
0.46
0.54
0.71
0.14
0.70
0.35
0.89
0.89
0.50
0.83
0.80
0.89 0.07
0.51
0.32
0.34
0.32
0.49
cott-buse-uncused-oll
0.68
0.57
0.43
0.01
0.00
0.34
0.76
0.75
0.26
0.61
0.34
0.86
0.84
0.43
0.79
0.75
0.89
0.01
0.56
0.32
0.36
0.15
0.49
bart-large uncased-oi ali
0.74
0.54
0.58
0.13
0.02
0.39
0.81
0.78
0.55
0.64
0.21
0.84
0.85
0.64
0.80
0.71
0.90
0.08
0.61
0.38
0.42
0.12
0.53
0.73
0.64
0.43
0.13
0.02
0,36
0.73
0.83
0.39
0.77
0.33
0.84
0.85
0.58
0.78
0.72
0.90
0.06
0.60
0.37
0.28
0.20
0.52
deberta-v3 v3-has----
0.76
0.50
0.44
0.04
0.68
0.83
0.37
0.78
0.46
0.90
0.91
0.93
0.12
0.34
0.38
0.29
0.55
deberta-v3 -3-large-nii
0.81
0.54
0.27
0.21
0.06
0.34
0.60
0.82
0.32
0.80
0.53
0.92
0.80
0,90
0.85
0.98
0.35
0.66
0.42
0.44
0.44
0.59
deberta-v3-large --are------gee-l-tttllttttt
0.83
0.64
0,27
0.25
0.06
0.69
0.84
0.44
0.82
0.28
0.93
0.93
0.79
0.84
0.98
0.24
0.40
0.42
0,41
0.60
modembert hase-nli
0.74
0.55
0.64
0.12
0.02
0.42
0.75
0.68
0.29
0.75
0.40
0.91
0.91
0.60
0.89
0.74
0.96
0.10
0.46
0.25
0.28
0.30
0.53
modernbert-large large-nii
0.76
0.42
0.44
0.04
0.09
0.37
0.59
0.81
0.45
0.72
0,48
0.93
0.92
0.54
0.91
0.86
0.98
0.21
0.63
0.36
0.30
0.30
0.55
0.71
0.45
0,34
0.05
0.06
0.41
0.67
0.80
0.42
0.73
0,25
0.93
0.92
0.65
0.91
0.87
0.98
0.21
0.60
0.42
0,37
0.31
0.55
Rerankers
ms-marco- we--marco-Miil.Aae M-166v2
0.40
0.62
0.47
0.30
0.07
0.39
0.50
0.51
0.45
0.47
0.30
0.68
0.71
0.28
0.62
0.58
0.65
0.24
0.61
0.05
0.20
0.18
0.42
gle-reranker- modernbert-Mee base
0.68
0,52
0.60
0.42
0.17
0.58
0.53
0.56
0.43
0.55
0.37
0.91
0.92
0.50
0.84
0.80
0.96
0.65
0.36
0.54
0.45
0.39
0.58
bge-reranket-base
0.63
0.47
0,30
0.40
0.16
0.47
0.41
0.55
0,27
0,57
0,41
0.66
0.79
0.35
0.69
0.57
0.65
0.48
0.50
0.43
0.30
0.29
0.47
hge reranker large
0.73
0,53
0,20
0.48
0.16
01.4
0.38
0.53
0.15
0.60
0.53
0.87
0.89
0.49
0.80
0.76
0.88
0.57
0.59
0.45
0.39
0.34
0.53
Qwen3-Rerande Reranker-O..ee
0.79
0,57
0.08
0.53
0.27
0.34
0.74
0.80
0,50
0.79
0.55
0.91
0.89
0.41
0.88
0.78
0.95
0.63
0,50
0.53
0.49
0.41
0.61
0.79
0,77
0.64
0.66
0,33
0.36
0.82
0.88
0,58
0.86
0.61
0.96
0.93
0.82
0.95
0.90
0.98
0.69
0.74
0.67
0.49
0.48
0.72
Embedding models
all-Minil MI-Minni.MM---- M-L-6v2
0.49
0.48
0.51
0.48
0.15
11.00
0.32
0.50
0,26
0,51
0.36
0.35
0.41
0.31
0.34
0.34
0.33
0.43
0.44
0.33
0.11
0.15
0.37
c5-base-v2
0.76
0,59
0,20
0.53
0.21
0.44
0.74
0.64
0.31
0.67
0.55
0.93
0.93
0.46
0.90
0.84
0.95
0.62
0.60
0.47
0.43
0.37
0.60
c5-large-v2
0.79
0,52
0,39
0.50
0.22
0.47
0.51
0.66
0.29
0.69
0.52
0.94
0.91
0.52
0.93
0.85
0.98
0.58
0.54
0.51
0.44
0.38
0.60
e5 mistral--71 serll--b-insttie ib-instruct
0.77
0.47
0.08
0.62
0.30
0.42
0.40
0.33
0.09
0.37
0.64
0.94
0.93
0.62
0.91
0.84
0.98
0.65
0.67
0.63
0.50
0.50
0.58
bge-base-en-vi.5
0.63
0.57
0.16
0.54
0.20
0.48
0.46
0.68
0.17
0.72
0.51
0.93
0,90
0.43
0.90
0.81
0.94
0.64
0.57
0.53
0.43
0.30
0.57
bge-large-en-vi.5
0.77
0.46
0.13
0.56
0,26
0,40
0.44
0.44
0.11
0.50
0.57
0.95
0.92
0.46
0.93
0.82
0.95
0.68
0.54
0.53
0.44
0.35
0.55
gte-base-en-vi.5
0.75
0.63
0,10
0.57
0,21
0.35
0.61
0.74
0,22
0.65
0.56
0.90
0.93
0.47
0.85
0.84
0.97
0.66
0.60
0.51
0.42
0.32
0.58
gle fee-large-en v1.5
0.74
0.47
0,21
0.55
0,28
0.40
0.75
0.82
0,38
0.82
0.56
0.95
0.94
0.93
0.63
0.57
0.40
0.34
0.62
gie-modembert-bbse
0.76
0.59
0.13
0.48
0.24
0.47
0.63
0.52
0.15
0.48
0.54
0.95
0.92
0.64
0.91
0.82
0.96
0.64
0.63
0.61
0.44
0.39
0.59
Owen3--Embeddiing-0.666
0.66
0.52
0.32
0.53
0.24
0.47
0.52
0.54
0.32
0.66
0.55
0.90
0.87
0.49
0.88
0.76
0.96
0.64
0.59
0.48
0.38
0.58
Qwen3 Embedding 8B
0.77
0.54
0.26
0.55
0.32
0.41
0.31
0.39
0.27
0.47
0.59
0.94
0.92
0.72
0.95
0.86
0.96
0.71
0.40
0.65
0.54
0.47
0.59
Instruction-tuned LLMs
gemma--3270m-it
0.18
0.54
0.26
0.03
0.00
0.44
0.47
0.49
0.20
0.51
0.03
0.46
0.48
0.21
0.47
0.47
0.43
0.00
0.38
0.00
0.06
0.01
0.28
gemma-3-Ib-i
0.36
0.48
0.44
0.16
0.01
0.39
0.50
0.55
0.21
0.52
0.17
0.52
0.68
0.25
0.56
0.57
0.53
0.10
0.52
0.08
0.13
0.16
0.36
IJama 3.2mm--3.2-BB--- Instruct
0.67
0.57
0.30
0.44
0.10
0.46
0.50
0.52
0.25
0.53
0.46
0.51
0.51
0.28
0.51
0.49
0.44
0.39
0.46
0.37
0.33
0.37
0.43
Qwen3-4B
0.82
0.69
0.64
0.55
0.17
0.34
0.85
0.89
0.67
0.87
0,52
0.94
0.92
0.66
0.92
0.87
0.98
0.37
0.45
0.38
0.43
0.32
0.65
Phi-4-mini-instree
0,57
0.58
0,26
0.46
0.15
0.47
0,53
0.53
0,25
0,56
0.47
0.56
0.55
0.23
0.56
0.53
0.52
0.42
0.41
0.28
0.30
0.30
0.43
Qwen3 8B
0.85
0.72
0.67
0.53
0.18
0.34
0.84
0.89
0.72
0.86
0.55
0.94
0.92
0.75
0.94
0.86
0.98
0.39
0.67
0.39
0.40
0.24
0.66
0.84
0.79
0.81
0.51
0,18
0.41
0.88
0.91
0.83
0.85
0.59
0.94
0.90
0.75
0.95
0.54
0.97
0.35
0.65
0.36
0.44
0.29
0.67
Table 5: Zero-shot classification results on BTZSC by dataset (macro-averaged F1). Abbreviations: AGN = AGNEWS, BF-Off / BF-Sex = BiasFrames (offensive / sex), CAPS CAPSOTU, MAN Manifesto, TT = TrueTeacher, WT-Ins /TT-Obs/WT-Ob WT-Obs / WT-Thr WTT-Ag = WikiToxic (insult / obscene / threat/ toxic aggregated), YT= Yahoo Topics, AmzPol = Amazon Polarity. AppR AppReviews. FPB = FinancialPhraseBank, RT = Rotten Yelp= = YelpReviews, B77 = Banking77,
Bnt = BiasFrames (intent), MASS = MASSIVE (intent), EmoD = EmotionDiary, Emp = EmppathtticDialogues.

---

Task
Topic / toxicity
Sentiment FPB
Intent
Emotion
Published as a conference paper at ICLR 2026
Model
AGN BE-OIT
BE-Sex CAPS
MAN
TT
WT-Ins
WT-Obs
WT-Thr WT-Agg
YT
AmzPol AppR
IMDB
RT
Yelp
177
BR-Int
MASS
Emol
Emp
Avg Ace
Base encoders
bett-laggeumcased
0.26
0.57
0.91
0.09
0.02
0.49
0.58
0.58
0.78
0.55
0.13
0.48
0.51
0.56
0.52
0.52
0.52
0.03
0.44
0.01
0.20
0.03
0.40
deherta-v3 large
0.28
0.58
0.12
0.01
0.04
0.51
0.65
0.64
0.86
0.58
0.11
0.49
0.51
0.40
0.49
0.50
0.52
0.03
0.47
0.00
0.20
0.03
0.36
ModemBBERT-larg
0.24
0.53
0.72
0.03
0.01
0.51
0.46
0.59
0.14
0.60
0.15
0.49
0.50
0.57
0.51
0.50
0.51
0.03
0.48
0.02
0.14
0.02
0.35
NLI cross-encwders
hart hart-large-mill
0.73
0.57
0.08
0.39
0.09
0.51
0.42
0.69
0.08
0.61
0.30
0.93
0.92
0.42
0.93
0.83
0.96
0.29
0.63
0.43
0.50
0.39
0.53
ali-roberta-buse
0.70
0.58
0.19
0.23
0.02
0.48
0.56
0.71
0.14
0.70
0.41
0.89
0.89
0.45
0.83
0.80
0.89
0.08
0.51
0.36
0.34
0.32
0,50
port-bse-umccsed-nll
0.69
0.57
0.53
0.02
0.02
0.49
0.76
0.77
0.29
0.65
0.38
0.86
0.84
0.40
0.79
0.76
0.89
0.02
0.56
0.28
0.39
0.15
0.50
large uncased-n nli
0.75
0.60
0.76
0.22
0.02
0.49
0.79
0.25
0.84
0.86
0.81
0.35
0.55
0.13
0.57
lortlarge-nncased-li--
0.73
0.67
0.53
0.24
0.02
0.49
0.73
0.83
0,50
0.78
0.42
0.84
0.85
0.59
0.79
0.73
0.07
0.60
0.37
0.30
0.21
0.55
deberta-v3-bass-nni
0.76
0.60
0.55
0,23
0.05
0.49
0.68
0.83
0.46
0.78
0.53
0.90
0.91
0.67
0.91
0.82
0.13
0.51
0.33
0.38
0,32
0.58
Jeberta-v3-large
0.81
0.63
0.30
0.23
0.11
0.50
0.61
0.82
0.37
0.80
0.60
0.92
0.93
0.82
0.90
0.85
0.98 0.36
0.67
0.44
0.49
0.46
0.62
deberta-3-large-nli--ine
0.83
0.67
0,30
0.27
0.09
0.51
0.69
0.84
0.57
0.82
0.37
0.93
0.93
0.82
0.93
0.84
0.98
0.24
0.70
0.42
0.44
0.42
0,62
modernhert base-nli
0.75
0.62
0.84
0.18
0.03
0.50
0.75
0.71
D.33
0.75
0.44
0.91
0.91
0.61
0.89
0.75
0.96
0.09
0.52
0.28
0.28
0.29
0,56
modernbert-large -lrg
0.76
0.59
0,55
0.09
0.10
0.52
0.61
0.81
0.59
0.73
0.54
0.93
0.92
0.70
0.91
0.86
0.98
0.24
0.63
0.39
0.28
0.31
0.59
0,72
0.60
0.40
0.13
0.08
0.51
0.67
0.80
0,54
0.74
0.29
0.93
0.92
0.73
0.91
0.87
0.98
0.22
0.60
0.44
0.37
0.34
0.58
Rerankers
92
mss -aa--aaroo----- no--marcoMMinLLMM MiniLM-L66-22
0.42
0.62
0.75
0.32
0.14
0.52
0.53
0.55
0.65
0.50
0.33
0.69
0.71
0.29
0.62
0.58
0.65
0.22
0,62
0.04
0.20
0.18
0.46
gte-reranker-mmodenee ceraanker-moddebee---beee
0,71
0.52
0.85
0.48
0.26
0.60
0.55
0.57
0.60
0.55
0.39
0.91
0.92
0.47
0.84
0.80
0.96
0.63
0.47
0.57
0.52
0.39
0.62
hge reranker- base
0.65
0.48
0.35
0.44
0.27
0.47
0.46
0.55
0.31
0.57
0.47
0.66
0.79
0.35
0.69
0.58
0.65
0.49
0.50
0.45
0.35
0.28
0.49
bge-reranker-larg large
0.74
0.58
0.21
0.52
0.32
0.45
0.44
0.56
0.16
0.62
0.59
0.87
0.89
0.48
0.80
0.76
0.88
0.56
0.60
0.47
0.41
0.34
0.56
0,80
0.64
0.08
0.59
0.40
0.50
0.75
0.81
0.70
0.79
0.61
0.91
0.89
0.38
0.88
0.78
0.95
0.62
0.56
0.52
0.55
0.42
0.64
Qwen3- Reranker-SS SB
0.80
0.78
0.82
0.68
0.46
0.52
0.82
0.88
0.78
0.96
0.93
0.84
0.95
0.90
0.67
0.72
0.56
0.49
0.76
Embedding models
all-MiniLM--Lóv2
0.50
0.50
0.77
0.53
0.30
0.49
0.41
0.50
0,30
0.53
0.38
0.49
0.53
0.42
0.49
0.50
0.49
0.44
0.53
0.34
0.11
0.17
0.44
e5 bases-ee v2
0.77
0.62
0.21
0.59
0.49
0.74
0.65
0.36
0.68
0.62
0.93
0.93
0.90
0.84
0.62
0.49
0.51
0.41
0.62
e5-large-v2
0.79
0.52
0.50
0.55
0.34
0.48
0.54
0.66
0.33
0.69
0.56
0.94
0.91
0.45
0.93
0.85
0.56
0.54
0.51
0.49
0.43
0.62
c5-mistral-7h-instuus
0.78
0.60
0.08
0.62
0.50
0,48
0.46
0.43
0.09
0.51
0.71
0.94
0.93
0.57
0.92
0.84
0.65
0.68
0.67
0.55
0.54
0.62
0.65
0.60
0.17
0.58
0.34
0.48
0.50
0.68
0.17
0.73
0.59
0,93
0.90
0.39
0.90
0.81
0.94
0.64
0.58
0.56
0.52
0.35
0.59
bge-large-en-v1.5
0.77
0.57
0.13
0.61
0.43
0.47
0.49
0.50
0.11
0.57
0.63
0,95
0.92
0.41
0.94
0.82
0.95
0.68
0.58
0.52
0.55
0.39
0.59
gle base-en-vll.
0.76
0.64
0.10
0.59
0.36
0.48
0.62
0.74
0.23
0.67
0.62
0.90
0.93
0.41
0.85
0.84
0.97 0.65
0.60
0.54
0.50
0.36
0.61
gte-large-en--1.55
0.75
0.48
0,22
0.59
0.39
0.45
0.75
0.82
0.47
0.82
0.62
0.95
0.91
0.47
0.94
0.87
0.93 0.63
0.56
0.58
0.45
0.36
0.64
gle modembeent base
0.76
0.63
0.13
0.52
0.38
0.47
0.63
0.55
0.16
0.55
0.60
0.95
0,92
0.64
0.91
0.82
0.96
0.63
0.63
0.65
0.50
0.40
0.61
Twen3-Embeddingee
0.68
0.56
0.38
0.55
0.40
0.47
0.55
0.55
0,38
0.67
0.60
0.90
0.87
0.45
0.88
0.76
0.96
0.64
0.53
0.63
0.53
0.40
0.61
Qwen F-mbeedddddnng-
0.78
0.64
0.30
0.52
0.47
0.51
0.41
0.47
0.30
0.56
0.64
0.94
0.92
0.69
0.95
0.86
0.96
0.70
0.56
0.71
0.59
0.50
0.64
Instruction-tuned LLMs
gemma-3 3 it
0.24
0.59
0.29
0.09
0.00
0.50
0.49
0.51
0.22
0.54
0.08
0.51
0.51
0.27
0.52
0.50
0.47
0.01
0.41
0.01
0.11
0.04
0.31
gemma.3.1b-it
0.39
0.52
0.58
0.20
0.46
0.52
0.57
0.56
0.19
0.59
0.69
0.25
0.60
0.14
0.09
0.19
0.22
0.40
Llama-3.2-3B- -nsttuuttt
0.68
0.60
0.35
0.50
0.50
0.50
0.53
0.29
0.56
0.50
0.54
0.54
0.30
0.54
0.51
0.48
0.41
0.47
0.37
0.46
0.40
0.46
Qwen3-4B
0.82
0.71
0.83
0.60
0.29
0.51
0.86
0.89
0.88
0.87
0.56
0.94
0.92
0.62
0.92
0.87
0.40
0.58
0.40
0.54
0.34
0.70
Phi-4-mini 4 mini mini-instruct instruct
0.56
0.64
0.28
0.45
0.30
0.50
0.55
0.55
0.28
0.60
0.45
0.62
0.60
0.22
0.61
0.58
0.60
0.43
0.44
0.31
0.34
0.31
0.47
Qwen3-8B
0.85
0.75
0.86
0.57
0.29
0.51
0.84
0.89
0.91
0.86
0.58
0.94
0.92
0.76
0.94
0.86
0.98
0.40
0.70
0.41
0.54
0.26
0.71
Haatral-Nemo-Instrtee
0.84
0.79
0.95
0.60
0.26
0.54
0.89
0.91
0.96
0.85
0.65
0.94
0.90
0.71
0.95
0.61
0.97
0.39
0.69
0.39
0.55
0.32
0.71
Table 6: Zero-shot classification results on BTZSC by dataset (micro-averaged accuracy). Abbreviations: AGN = AGNEWS, BF-Of / BF-Sex = BiasFrames (offenssive/ / sex), CAPS = CAPSOTU, MAN = Manifesto, TT = Teachea WT-Ins / WT-Obs / WT-Thr WT-Agg = WikiToxic (insult obscene tthratt toxic aggregated), YT = Yahoo Topies, AmzPol = Amazon Polarity. AppR = AppReviews, FPB = FinancialPhraseBank. RT = RottenTomatoes, Yelp YelpReviews, B77 = Banking77, BF-Int = BiasFrames (intent), MASS = MASSIVE (intent), EmoD = EmotionDiary, Emp = EmpatheticDDaalouues.

---

Task
Topic / toxicity
Sentiment
Intent
Emotion
Published as a conference paper at ICLR 2026
Model
AGN BF-Off BF-Sex CAPS
MAN
TT
WT-Ins WI-Obs WT-Thr WT-AgE
YT
AmzPol AppR
FPB
IMDB
RT
Yelp
B77
BF-Int MASS
EmoD
Emp
Avg R
Base encuders
bert hert-large- aattlaage-uncseed uncased
0.26
0.53
0.49
0.10
0.02
0.50
0.52
0.58
0.55
0.55
0.12
0.47
0.51
0.33
0.51
0.52
0.51
0.03
0.46
0.03
0.18
0.03
0.35
deherta-y3 -3- large
0.27
0.53
0.50
0.04
0.02
0.50
0.58
0.58
0.52
0.57
0.09
0,50
0.51
0.39
0.50
0.50
0.51
0.03
0.49
0.00
0.18
0.03
0.36
ModemBERRT-aan
0.25
0.55
0.50
0.07
0.01
0.50
0.47
0.59
0.49
0.61
0.14
0,50
0.50
0.33
0.50
0.50
0.52
0.04
0.52
0.03
0.18
0.02
0.36
NLI cross-encoders
burt-large-mnli
0.73
0.50
0.51
0.37
0.10
0.52
0.52
0.73
0.52
0.62
0.28
0.93
0.92
0.67
0.93
0.83
0.96
0.30
0,61
0.43
0.48
0.38
0.58
nli ofirrherta--ban base
0.71
0.53
0.56
0.18
0.02
0.48
0.63
0.72
0.55
0.71
0.36
11.89
0.89
0.65
0.83
0.80
0.89
0.07
0.53
0.33
0.46
0.32
0.55
bertbase-unnassee
0.69
0.58
0.73
0.05
0.02
0.50
0.79
0.75
0.62
0.64
0.34
0,86
0.84
0.60
0.79
0.76
0.89
0.01
0.57
0.31
0.42
0.15
0.54
uit-lange-uncuucc--lii
0.75
0,56
0.82
0.17
0.04
0.50
0.84
0.78
0.85
0.67
0.22
0.84
0.85
0.76
0.81
0.72
0.90
0.08
0.61
0.36
0.43
0.13
0.58
oott-lag--umcusc---ll-ippp-.
0.74
0.64
0.73
0.19
0.04
0.50
0.77
0.84
0.74
0.78
0.37
0.84
0.85
0.71
0.78
0.73
0.90
0.07
0.61
0.43
0.35
0.20
0.58
deherta-v3
-h-rta--3-hhss--n (3-hase-nli
0.77
0.55
0.74
0.19
0.08
0.50
0.73
0.83
0.72
0.79
0..77
0.90
0.91
0.80
0.91
0.82
0.93
0.13
0.54
0.34
0.42
0.31
0.61
deberta-v3- -a-legeill
0.82
0.59
0.63
0,23
0.11
0.50
0.68
0.84
0.67
0.80
0.54
0.92
0.93
0.84
0.90
0.85
0.98
0.38
0.69
0.44
0.48
0.45
0.65
0.83
0.64
0.63
0,30
0.09
0.52
0.74
0.85
0.77
0.82
0.33
0.93
0.93
0.80
0.93
0.84
0.98
0,26
0.70
0.44
0.47
0.42
0.65
modernhert hase nli
0.75
0.58
0.85
0.19
0.05
0.50
0.78
0.68
0.75
0.40
0.91
0.64
0.89
0.75
0.96
0.09
0.55
0.26
0.34
0.28
0.58
modernbert-large largo---l--
0.76
0.52
0.75
0.08
0.14
0.51
0.67
0.81
0.79
0.74
0.49
0.93
0.92
0.51
0.91
0.86
0.98
0,24
0.64
0.38
0.35
0.30
0.60
modernbert-lee baaiboer---aagell--iii-eee
0.72
0.54
0.67
0.08
0.08
0.51
0.73
0.81
0.75
0.75
0.26
0.93
0.92
0.61
0.91
0.87
0.98
0.22
0.61
0.45
0.42
0.32
0.60
Rerankers
L7
mss mis-marco-MimL.M. M-L-L2
0.42
0.64
0.49
0.36
0.10
0.51
0.50
0.52
0.60
0.50
0.30
0.69
0.71
0.49
0.63
0.58
0.66
0.22
0.61
0.06
0.27
0.18
0.46
gte-reranker-ne moonkk-rmmmeebbee--bbeeee
0.71
0.52
0.69
0.48
0.21
0.60
0.54
0.56
0.55
0.39
0.91
0.92
0.65
0.84
0.80
0.96
0.67
0.50
0.57
0.48
0.39
0.62
age-reranker-base
0.66
0.47
0.59
0.52
0.19
0.47
0.54
0.57
0.60
0.58
0.42
0.66
0.79
0.50
0.70
0.58
0.65
0,51
0.50
0.53
0.34
0.27
0.53
rerrrrrnkeer--rge
0.74
0.55
0.58
0.57
0.17
0.45
0.53
0.61
0.56
0.62
0.53
0.87
0.89
0.58
0.81
0.76
0.88
0,58
0.59
0.57
0.48
0.33
0.60
Qwen3- Owen3-Rerankeer-tee Reranker-O.66
0.79
0.60
0.50
0.59
0.31
0.50
0.75
0.80
0.75
0.79
0.54
0.91
0.89
0.62
0.88
0.78
0,95
0.66
0.54
0.61
0.52
0.42
0.67
Qwen3-Reranker Reranker-8B
0.80
0.76
0.88
0.69
0.35
0.51
0.85
0.88
0.88
0.86
0.60
0.96
0.93
0.80
0.95
0.90
0.98
0.72
0.74
0.74
0.51
0.49
0.76
Embedding models
all-MiniLM--L6-22
0.51
0.55
0.58
0.50
0.17
0.49
0.50
0.51
0.58
0.54
0.35
0.50
0.54
0.32
0.51
0.50
0.50
0.44
0.56
0.39
0.23
0.16
0.45
c5-base-v2
0.77
0.59
0.51
0.64
0.25
0.49
0.75
0.68
0.63
0.68
0.56
0.93
0.93
0.60
0.90
0.84
0.95
0.65
0.61
0.58
0.45
0.41
0.65
e5-lurge-v2
0.79
0.52
0.57
0.56
0.25
0.48
0.61
0.70
0.64
0.70
0.56
0.94
0.91
0.68
0.93
0.85
0.98
0.59
0.55
0.60
0.50
0.42
0.65
e5 <5-mistral--b-insteee 7b-instret
0.78
0.54
0.51
0.69
0.32
0.48
0.55
0.51
0.53
0.52
0.64
0.94
0.93
0.75
0.91
0.84
0.98
0.69
0.67
0.70
0.54
0.54
0.66
bge-base-en-v1.5
0.65
0.57
0,51
0.64
0.23
0.48
0.58
0.72
0.56
0.73
0.59
0.93
0.90
0.65
0.90
0.81
0,94
0.67
0.57
0.64
0.47
0.34
0.64
hge-lange-en-y1155
0.77
0.52
0.52
0.66
0.31
0.48
0.57
0.56
0.51
0.58
0.58
0.95
0.92
0.64
0.93
0.82
0.95
0,70
0.56
0.62
0.48
0.39
0.64
1.--base-en-v1.5
0.76
0.63
0,51
0.64
0,25
0.49
0.68
0.77
0.59
0.68
0.56
0.90
0.93
0.63
0.85
0.84
0.97
0.68
0,60
0.61
0.45
0.36
0.65
ste-large-en-v/
0.75
0.50
0.58
0.64
0.33
0.45
0.79
0.84
0.72
0.83
0.55
0.95
0.91
0.69
0.94
0.87
0.93
0.65
0.57
0.66
0.45
0.36
0.68
ste-moderbber-buse
0.76
0.60
0.52
0.61
0.29
0.48
0.69
0.60
0.56
0.56
0.54
0.95
0.92
0.66
0.91
0.82
0.96
0.66
0.63
0.69
0.49
0.40
0.65
Feenn--Enedddig--- 0.66B
0.68
0.53
0.56
0.65
0.28
0.47
0.62
0.59
0.68
0.68
0.54
0.90
0.87
0.65
0.88
0.76
0.96
0.67
0.50
0.67
0.55
0.41
0.64
Qwen3- Baana--Ebeediing-- SB
0.78
0.59
0.46
0.65
0.40
0.50
0.51
0.54
0.63
0.57
0.58
0.94
0.92
0.78
0.95
0.86
0.96
0.74
0.52
0.72
0.60
0.50
0.67
Instruction-tuned L.Mss
gemmaa--227m-it
0.23
0.55
0.48
0.05
0.02
0.50
0.54
0.56
0.47
0.55
0.07
0,50
0.51
0.33
0.52
0.50
0.47
0.01
0.43
0.01
0.14
0.04
0.34
gemma-3-1b-it
0.38
0.49
0.65
0.15
0.03
0.46
0.59
0.62
0.56
0.57
0.23
0.58
0.69
0.46
0.60
0.60
0.59
0.15
0.54
0.11
0.20
0.21
0.43
Llama-3.2-3B- 1-mm-----3B-nnstruett
0.68
0.57
0.52
0.48
0.12
0.49
0.55
0.57
0.50
0.56
0.46
0.54
0.54
0.37
0.53
0.51
0.47
0.44
0.48
0.40
0.35
0.39
0.48
Qwen3-4B
0.82
0.69
0.86
0.58
0.21
0.50
0.87
0.89
0.91
0.87
0.51
0.94
0.92
0.74
0.92
0.87
0.98
0.42
0.54
0.43
0.40
0.34
0.69
Phi-4-mini-instrue
0.56
0.60
0.51
0.48
0.17
0.50
0.62
0.60
0.60
0.61
0.46
0.61
0.60
0.41
0.60
0.58
0.59
0.45
0.46
0.29
0.32
0.32
0.50
Qwen3-8B
0.85
0.72
0.90
0.55
0.22
0.50
0.86
0.89
0.92
0.86
0.53
0.94
0.92
0.84
0.94
0.86
0.98
0.43
0.68
0.43
0.39
0.26
0.70
Mistral Nemo- Instruct-2407
0.84
0.79
0.85
0.52
0.20
0.53
0.85
0.60
0.90
0.79
0.95
0.61
0.97
0.40
0.67
0.40
0.43
0.30
0.69
Table 7: Zero-shot classification results on BTZSC by dataset (macro-averaged recall). Abbreviations: AGN=AGNENTH BF-Off BF-Sex BiasFrames (offensive
/ sexx) CAPS CAPSOTU, MAN = Manifesto, Tor=Truueeeee Teacher, WT- WT-Obs WT- SETTThr/WT--AA WT-Agg WikiTToxic (insult/ obscene / threat / toxic aggregated), YT= Yahoo Topics, AmzPol= = Amazon Polarity, AppR = AppReviews, FPB = FinancialPPrraseBBnn, RT Rotten'ommatos, Yelp = YelpReviews, B77 Banking77, BF- = BiasFrames (intent), MASS- = MASSIVE (intent), EmoD = EmotionDiary, Emp EmpatheticcDialogues

---

Task
Topic/ / toxicity
Sentiment
Intent
Emotion
Published as a conference paper at ICLR 2026
Model
AGN BF-Off BF-Sex CAPS
MAN
TT
WT-Ins
WT-Obs WT-Thr WT-Agg
YT
AmzPol AppR
FPB
IMDB
RT
Yelp
B77
BF-Int MASS
EmoD
Emp
Avg P
Base encuders
bett-lage-uncased
0.18
0.55
0.47
0.04
0.03
0.25
0.53
0.58
0.51
0.56
0.11
0.45
0.67
0.38
0.55
0.53
0.58
0.02
0.45
0.00
0.19
0.02
0.35
deherta-y3 y3- large
0.38
0.56
0.50
0.01
0.02
0.25
0..6
0.66
0.51
0.62
0.04
0.24
0.58
0.32
0.24
0.53
0.58
0.02
0.49
0.00
0.10
0.00
0.33
ModemBERT-large
0.06
0.55
0.50
0.02
0.02
0.53
0.47
0.58
0.50
0.61
0.07
0.24
0.49
0.29
0.26
0.50
0.56
0.05
0.59
0.04
0.07
0.01
0.32
NLI cross-encaders
burt-laree-mlli
0.77
0.78
0.53
0.59
0.22
0.52
0.62
0.76
0.52
0.74
0.53
0.93
0.92
0.64
0.93
0.84
0.96
0.47
0.65
0.56
0,48
0.48
0.66
nli nfi-roherta-hee base
0.72
0.58
0.53
0.26
0.09
0.48
11.70
0.72
0.52
0.73
0.46
0.89
0.889
0.61
0.83
0.80
0.89
0.13
0.53
0.47
0.47
0.43
0.58
bot--ase-uncased-aa
0.73
0.58
0.55
0.06
0.01
0.45
0.78
0.77
0.53
0.71
0.45
0.87
0.84
0.61
0.80
0.77
0.89
0.03
0.57
0.54
0.45
0.36
0.56
lelllaaggnaaeeeee -lii
0.76
0.59
0.59
0.32
0.07
0,50
0.82
0.80
0.57
0.75
0.49
0.85
0.86
0.63
0.84
0.76
0.90
0.17
0.61
0.53
0.46
0.33
0.60
bott-lage-uncacc--ll--ii---
0.75
0.68
0.55
0.18
0.06
0.48
0.79
0.83
0.54
0.79
0.41
0.86
0.85
0.61
0.82
0.77
0.90
0.11
0.61
0.45
0.40
0.34
0.58
deherta-v3 v3-hase-nli
0.76
0.61
0.55
01.38
0.13
0.39
0.77
0.83
0.54
0.79
0.51
0.90
0.91
0.68
0.91
0.82
0.93
0.23
0.58
0.49
0.46
0.46
0.62
deberta-v3- -allageeill
0.81
0.70
0.54
0.41
0.11
0.66
0.74
0.83
0.53
0.81
0.56
0.93
0.93
0.78
0.91
0.86
0.98
0.45
0.71
0.57
0.47
0.56
0.67
deberta-vS v3- -3---r------------- --lagg-nl--trpplae
0.83
0.67
0.54
0.48
0.18
0.56
0.78
0.84
0.55
0.85
0.39
0.93
0.93
0.78
0.93
0.84
0.98
0.35
0.70
0.55
0.50
0.54
0.67
moodembert-base-ni
0.77
0.64
0.62
0.29
0.08
0.51
0.78
0.71
0.53
0.75
0.52
0.91
0.91
0.63
0.89
0.80
0.96
0.20
0.60
0.39
0.44
0.44
0.61
modernhert- larrgeeer---rg--it
0.68
0.56
0.18
0.62
0.81
0.55
0.78
0.53
0.93
0.76
0.92
0.88
0.98
0.29
0.47
0.47
0.47
modernbert-large taaeeenbbrr--rg-------i---e
0.77
0.70
0.54
0.17
0.21
0.52
0.76
0.81
0.54
0.80
0.41
0.93
0.92
0.72
0.92
0.88
0.98
0.36
0.63
0.53
0.49
0:45
0.64
Rerankers
38
0.50
0.64
0.50
0.36
0.13
0.57
0,50
0.52
0.52
0.49
0.33
0.71
0.72
0.43
0.64
0.58
0.67
0.44
0.61
0.14
0.33
0.35
0.49
gte-reranker-non maaankerr-mmddebbe---beeee
0.76
0.52
0.58
0.46
0.61
0.54
0.56
0.52
0.55
0.56
0.92
0.93
0.62
0.86
0.83
0.96
0.68
0.53
0.58
0.45
0.49
0.63
bge-reranker-buse
0.67
0.47
0.52
0.41
0.18
0.47
0.61
0.58
0.52
0.58
0.45
0.66
0.79
0.44
0.70
0.58
0.65
0.57
0.50
0.44
0.32
0.40
0.52
bge- agge-reannee--iee large
0.77
0.56
0.53
0.51
0.45
0.62
0.68
0.64
0.54
0.87
0.89
0.51
0.81
0.76
0.89
0.66
0.44
0.44
0.49
0.61
owen3-Reranker-O.66
0.82
0.69
0.51
0.55
0.31
0.25
0.74
0.80
0.55
0.79
0.58
0.91
0.89
0.63
0.88
0.78
0.95
0.66
0.57
0.56
0.49
0.47
0.65
Qwen Swwn3-RRerndee Reranker-88
0.84
0.78
0.62
0.71
0.40
0.76
0.84
0.88
0.58
0.86
0.65
0.96
0.93
0.85
0.95
0.91
0.98
0.74
0.79
0.71
0.50
0.58
0.76
Embedding models
all Mi.MMniMMMM. M-L.6-2
0.63
0.57
0.53
0.52
0.19
0.46
0.51
0.52
0.55
0.55
0.35
0.74
0.46
0.24
0.56
0.42
0.46
0.36
e5-hase-v2
0.78
0.61
0.50
0.51
0.25
0.49
0.71
0.53
0.70
0.57
0.93
0.93
0.53
0.90
0.84
0.95
0.66
0.50
0.45
0.50
e5- <5-large-v2
0.80
0.52
0.52
0.52
0.28
0.48
0.69
0.73
0.53
0.72
0.57
0.94
0.91
0.65
0.93
0.85
0.98
0.70
0.55
0.47
0.48
0.65
55-mistrul-7-iinstta
0.82
0.68
0.53
0.62
0.36
0.46
0.70
0.66
0.52
0.72
0.66
0.94
0.93
0.67
0.92
0.84
0.98
0.68
0.65
0.53
0.59
0.69
bge- nge-hase-ene en-51.5
0.68
0.59
0.51
0.53
0.25
0.48
0.69
0.74
0.52
0.75
0.56
0.93
0.91
0.60
0.90
0.82
0.95
0.68
0.58
0.54
0.50
0.40
0.64
bge-lurge-en-v1.5
0.77
0.54
0.52
0.55
0.29
0.45
0.70
0.68
0.51
0.68
0.60
0.95
0.92
0,61
0.93
0.83
0.95
0.74
0.58
0.54
0,51
0.48
0.65
b--bb----n-v..5
0.76
0.63
0.52
0.58
0.24
0.43
0.73
0.78
0.52
0.74
0.58
0.91
0.93
0.60
0.88
0.85
0.97
0,68
0.60
0.54
0.43
0.37
0.65
gte-large-en--1.55
0.78
0,51
0.53
0.56
0.36
0.43
0.80
0.83
0.54
0.85
0.59
0.95
0.91
0,57
0.94
0.88
0.94
0,68
0.57
0.59
0.47
0.43
0.67
gle modernhent base
0.76
0.63
0.51
0.49
0.29
0.48
0.73
0.67
0.52
0.67
0.56
0.95
0.92
0.67
0.91
0.83
0.96
0.69
0.63
0.62
0.48
0.45
0.66
Oweal-Embedddigg0..6
0.68
0.54
0.52
0.54
0.28
0.47
0.71
0.61
0.53
0.74
0.57
0.90
0.87
0.60
0.89
0.77
0.96
0.68
0,51
0.59
0.48
0.43
0.63
0.81
0.72
0.49
0.59
0.35
0.51
0.70
0.71
0.53
0.76
0.64
0.94
0.92
0.72
0.95
0.86
0.96
0.76
0.71
0.65
0,55
0.55
0.70
Instruction-tuned LLMs
comma-3-270m-it
0.25
0.57
01.49
0.014
0.00
0.50
11.56
0.58
0.49
0.57
0.12
0.50
0.52
0.36
0.53
0.50
0..4
0.00
0.41
0.00
0.04
0.01
0.34
gemmu-331h-it
0.50
0.49
0.53
0.29
0.02
0.43
11655
0.66
0.52
0.62
0.30
0.70
0.72
0.43
0.67
0.64
0.71
0.18
0.55
0.20
0.20
0.24
0.46
Llama-3.2-3B-Innse
0.73
0.59
0.50
0.49
0.24
0.49
0,56
0.58
0.50
0.58
0.51
0.55
0.55
0.36
0.54
0.51
0.46
0.54
0.48
0.46
0.42
0.48
0.51
Qwen3-4B
0.85
0.72
0.62
0.62
0.22
0.75
0.86
0.89
0.63
0.87
0.64
0.95
0.92
0.71
0.93
0.87
0.98
0.48
0.69
0.46
0.57
0.40
0.71
Phi-4 1hi-4-mini-insteee
0.74
0.67
0.50
0.55
0.24
0.50
0.70
0.63
0.52
0.69
0.58
0.71
0.70
0.38
0.69
0.66
0.71
0.53
0.44
0.40
0.39
0.41
0.56
Qwen3-8B
0.86
0.79
0.64
0.61
0.23
0.25
0.85
0.89
0.66
0.87
0.64
0.95
0.93
0.73
0.94
0.88
0.98
0.52
0.73
0.48
0.56
0.42
0.70
Mistral-Nent Newoo-nnttuet--222222 Instruct-2407
0.85
0.78
0.79
0.61
0.25
0.70
0.88
0.91
0.77
0.85
0.64
0.94
0.90
0.79
0.95
0.77
0.97
0.45
0.76
0.47
0.55
0.40
0.73
Table 8: Zero-shot classification results on BTZSC by dataset (macro-averaged precision). Abbreviations: AGN = AGNEWS, BF-Off BF-Sex = BiasFrames
(offensive/ / sex), CAPS- CAPSOTU, MAN Manifesto, TT'= = WT-lIns/ / WT-Obss / WT: / = WikiToxic (insult/ / obscene / /threat / toxic aggregated), YT = Yahoo Topics, AmzPol = Amazon Polarity, AppR = AppReviews, FPB- = FinancialPhraseBBnnk RT = Rotten Tomatoess Yelp = YelpReviews, B77 Banking77, Int BiasFrames (intent), MASS MASSIVE (intent), EmoD EmotionDiary, Emp = EmpatheticlDialogues.

---

Published as a conference paper at ICLR 2026
E COMPARISON WITH MTEB
To assess whether our conclusions depend on the specific dataset composition of BTZSC, we evaluate the same set of models on the English classification tasks from MTEB v2 (nevoldsen et al. 2025) (Amazon Counterfactual, MASSIVE Sentiment, MTOP Domain, ToxieConversations, IMDB, TweetSentiment Extraction, Banking77. MASSIVE Intent). Table/rreports macro-F, scores, Table TO accuracies, Table 11 recall, and Table T2 precision. We treat BTZSC and MTEB two independent evaluators that induce rankings over the same models and compare their behavior both in terms of average performance and the resulting rank orderings.
Global agreement. Across all models, m on BTZSC and MTEB strongly aligned. The Kendall rank correlation coefficient (Kendall & Gibbons, 1990) between the BTZSC and MTEB Avg-Fi rankings is high and positive (T 0.69, p z 1.3 10-"), indicating substantial agreement in how the two benchmarks order models. The family-wise picture mirrors our main BTZSC re- sults: base encoders perform worst, NLI cross-encoders substantially improve over them, modern rerankers and instruction eestruction-tuned LLMs are competitive, and contemporary embedding models attain the highest average macro-F, across both benchmarks. The top reranker, @wen Rer-RRennnk--BB is the single best model on both BTZSC and MTEB, while smaller rerankers such as 0.68 already outperform all NLJ cross-encoders in macro-Fi on both suites.
Rank consistency within model families. When restricting the correlation analysis to individual families, we still observe substantial agreement. For NLI cross-encoders, the BTZSC-MTEB rank correlation is = 0.64 (p a 0.0057), showing that models that are strong on BTZSC tend to remain strong on MTEB. Rerankers show almost perfect concordance (T R 1.0, p 0.0028), with both benchmarks inducing essentially the same ordering from the older ms-marco-MiniLM. Instruction-tume
L6-v2 up to (T 0.62, p & 0.069), reflecting consistent picture where very small LLMs perform poorly, and 4-8B models are competitive but do not surpass the best reranker. For embedding models, the correlation is positive but more moderate (r = 0.31, p 0.22): both benchmarks clearly favour modern embeddings (e5, BGE, GTE, Qwen-Embeddingg) over older baselines, but the fine-grained ordering within this family is somewhat benchmark-dependeen.
LLMs also exhibit sizable positive correlation
Embedding models and dataset composition. The embedding family illustrates well how dataset mix shapes absolute scores while leaving the main qualitative conclusions intact. Averaged over the eleven embedding models evaluated on both benchmarks, the family-level macro-F, 0.57 on BTZSC and 0.63 on MTEB, i.e. MTEB is systematically more "forgiving" to embeddings by roughly six F, points. This gap is largely explained by task composition.
On BTZSC, embedding models achieve mean task-wise macro-Fi of 0.47 topic classification, 0.80 on sentiment, 0.57 on intent, and 0.39 on emotion. On MTEB, grouping the eight datasets into sentiment-like tasks (Amazon Counterfactual, MASSIVE Sentiment, IMDB, TweetSentiment Extraction), intent-like tasks (Banking77, MASSIVE Intent), and topic-like tasks (MTOP Domain, ToxieConvessationns) the same embedding models reach on average 0.59 macro-F, sentiment, 0.58 on intent, and 0.74 on topic. Thus, relative to BTZSC, MTEB exposes embeddings to (i) much easier topic-style problems (mid 0.70s vs. mid 0.40s) and (ii) no emotion tasks all. Emo- tion classification is consistently difficult for all families on BTZSC (embeddings around 0.39; NLI cross-encoders and rerankers around 0.33-0.37), and the absence of this label family in MTEB re- moves a systematic downward pull on the macro averages. Conversely. MTEB's 's topic-style datasets (MTOP Domain and ToxicConversations appear particularly well aligned with embedding- based semantic similarity, yielding high scores that boost the family's average.
Importantly, BTZSC does not uniquely penalise embeddings: on BTZSC they are the strongest family on intent and emotion and competitive on topic and sentiment; MTEB simply provides a task mix-easier topic-style classification, no emotion-in which their strengths are accentuated. The fact that modern embeddings outperform NLI cross-encoders on average and remain few points below the best reranker holds on both BTZSC and MTEB. What differs is primarily the absolute level which they plateau (high-0.50s on BTZSC vs. low- to mid-0.60s on MTEB) and the precise ordering among closely matched embedding architectures. These observations indicate that BTZSC not only yields findings that are consistent with MTEBs classification suite, but also
29

---

Published as a conference paper at ICLR 2026
provides richer testbed for more nuanced performance analysis across tasks and domains, thanks to its explicit coverage of sentiment, topic, intent, and emotion with varying granularities.
Model
AmzCI
MASS-S
MTOP-D
ToxieConv
IMDB
TweetSentExt
B77
MASS-I
Arg F1
Base encoders
bert-large-uncaseel
0.03
0.02
0.00
0.20
deberta v3 larlerge
0.51
0.02
0.07
0.49
0.33
0.31
0.01
0.00
0.22
ModemBERI-lang
0.45
0.07
0.11
0.43
0.34
0.16
0.02
0.02
0.20
NLI cross-encoders
bart bart-larg--unli
0.15
0.54
0.84
0.50
0.93
0.54
0.28
0.41
0.52
roberta base
0.25
sott-bae-onncsed--il
0.20
0.23
0.64
0.52
0.79
0.59
0.01
0.32
0.41
tot-laage-umeesod-all
0.40
0.42
0.08
0.38
Nett-larg-unncssd-alll
0.38
0.31
0.49
0.53
0.78
0.61
0.06
0.37
0.44
deberta-v v3. base nli
0.15
0.48
0.68
0.91
0.12
0.34
0.49
Saberta-v3-large--nn
0.16
0.51
0.74
0.90
0.62
0.35
0,42
0.55
Saberta-33-large-nll-tte
0.34
0.56
0.49
0.72
0.93
0,58
0.24
0.40
0.53
moderbbert-uue baseett--bbbb-llle
0.35
0.25
modemberr large uli
0.42
0.42
0.59
0.69
0.91
0.64
0.21
0.36
0.53
modembeert trrppeli-trpplee
0.50
Rerankers
ma See-marro-MMilin MiniLM-L6-x
0,53
0.08
0.19
0,53
0.62
0,32
0.24
0.05
0.32
gle reranker moderbbert bass
0.65
hge-reranker-Be reranker-huse
0.39
0.44
0.73
0.56
0.69
0.53
0.48
0.43
0.53
bge-reranker-large
0.48
0.57
0.45
Qwen Me--RRaanke----GG
0.26
0.66
0.79
0.64
0.88
0.51
0.63
0.53
0.61
Qwen Reranker
0.46
0.77
0,80
0.74
0.95
0.62
0.69
0.67
0.71
Embedding models
all-MiniLM-LG-x2
0.15
0.48
0.63
0.42
0.34
0.40
0.43
0.33
0.40
c5-base-v2
0.55
0.81
0.90
0.62
0.47
0.61
large-v2
0.57
0.58
0,80
0.57
0.93
0.51
0.58
0.51
0.63
R.-mistal-7b-insttme
0.23
0.70
0.87
0.64
0.91
0.60
0.65
0.63
0.65
bge-base-en-vL.5
0.47
0.54
0.64
0.53
0.64
bge-large-ea--1.5
0,39
0.56
0.86
0.69
0.93
0.57
0.68
0.53
0.65
gle- ee--buse-en-v1.5
gte-large-en- en-v1_5
D.34
0.62
0.85
0,81
0.94
0.61
0,63
0.57
0.67
gle-modernbeet--uug
0.60
0.86
0.74
0.64
0.61
0.67
Qwen3 Embeddiin 0.6B
0.50
0.57
0.85
0.69
0.88
0.55
0.64
0.59
0.66
Qwen 3aad3--meddddig88
0.23
0.70
0.91
0.72
0.95
0.50
0.71
0.65
0.67
Instruction-tuned
gemma-3-277mm 3-270miit
0.34
0.03
0.05
0.47
0.47
0.24
0.00
0.00
0.20
gemma-3--1b
0.18
0.56
0.10
0.08
0.30
JJama--3224 3.2tretlntttuuet
0.35
0.53
0.55
0.50
0.51
0.36
0.39
0.37
0.44
Qwen3-4B
0.80
0.65
0.78
0.46
0.92
0.61
0.37
0.38
0.62
Pu--4-miniiimstteet
0.53
0.44
Qwen 8B
0.66
0.66
0.82
0.46
0.94
0.62
0.39
0.39
0.62
WistallNemmo-Insreee-2-0
0.64
0.35
0.36
Table 9: Zero-shot classification results (macro-averaged F1) eight English classification datasets from MTEB v2 plus their average (Avg FI), Bold denotes the best and underlining the second-best score each column. Best model in each family is underlined.
30

---

Published as a conference paper at ICLR 2026
ToxicCony
TweetSE
Avg Acc
Base eucoders
bert- lurge-uncased
0.25
0.05
0.15
0.53
0.52
0.41
0.03
0.01
0.25
deberta-v3-lurg ModemmBERT-large
0.03
0.49
0.32
0.03
0.00
0.83
0.09
0.13
0.50
0.51
0.33
0.03
0.02
0,30
NLI cross-encoders
bart- large-mnli
0.18
0.52
0.84
0.57
0.93
0.62
0.29
0.43
0.55
nlirrobeta--buse
0.83
0.08
bert- unse--ucassd-nl
0.21
0.21
0.62
0.58
0.79
0.64
0.02
0.28
0.42
bert-lune cett-urge-uncase--in
0.43
0.38
0.41
0.55
0.81
0.65
0.08
0.35
0.46
hert- hert-large-ancee www.larre-nccaeedeeeee nli-triplet
jeberta-v3-base-nll
0.17
0.49
0.65
0.69
0.91
0.63
0.13
0.33
0.50
deherta large uli
0.18
0.54
0.75
0.73
0.90
0.67
0.36
0.44
leberta-v3-lurge--n purge-nli-triplet
0.93
modernbert base-nll
0.68
0.36
0.47
0.68
0.89
0.62
0.09
0.28
0.51
modernbert-large-nll
0.43
0.40
0.59
0.69
0.91
0.66
0.24
0.39
0.54
modernbert warpubbeerrrrrreellle- -aar--aagg----ee triple
0.40
0.91
0.68
0.22
Rerankers
ms marco Caaarro-MinilMM MiniLM--L6. M-
0.76
0.08
0,23
0.53
0.62
0.42
0.22
0.04
0.36
gte-reranker mandukrrrmmddeeeer---bs
0.54
0.84
0.62
0.63
age-reranker-base
0.42
0.46
0.77
0.57
0.69
0.59
0.49
0.45
0.55
bge-reranker-large
0,50
0.50
0.82
0.64
0.80
0.61
0.56
0.47
0.61
Owen3- R--Reannkee---6B
0.26
0.88
0.62
0.62
0.47
0.75
0.81
0.74
0,95
0.65
0.67
0.72
0.72
Embedding models
Mll-Mini.M--L6.
0.17
0.47
0.49
0.49
0.40
0.44
0.34
0.43
c5-base-v2
0.33
0.55
0.84
0.65
0.90
0.60
0.62
0.49
0.62
e5- largee-2
0.66
0.58
0.83
0.61
0.93
0.59
0.56
0.51
0.66
0.71
0.92
0.64
hge-base-en-via
0.51
0.54
0.83
0.64
0.90
0.61
0.64
0.56
0.66
bge-turge-cn-y1.5
0,39
0.58
0.88
0.69
0.94
0.62
0.68
0.52
0.66
gle-base-en-v1.55
0.85
gte-large-en--l.55
0.34
0.63
0.86
0.81
0.94
0.66
0.63
0.58
0.68
gic-modernberi-bace
0,32
0.61
0.88
0.74
0.91
0.67
0.63
0.65
0.68
Owen3-EEneedding-O...
0.88
0.64
Qwen3 Feeenddin 8B
0.24
0.69
0.91
0.73
0.95
0.53
0.70
0.71
0.68
Instruction-tuned LLMs
gemma-3-270mnii
0.52
0.32
0.01
0.01
gemmu-3-lbbit
0.33
0.20
0.27
0.54
0.61
0.43
0.14
0.09
0.33
Llama- 3.2 2- Instruct
0.35
0.54
0.55
0.52
0.54
01.44
0.41
0.37
0.46
Qwen3-4B
Phi-4-mini-imstret
0.35
0.51
0.52
0.61
0.44
0.43
0.31
0.46
Qwen3-8B
0.71
0.65
0.82
0.56
0.94
0.67
0.40
0.41
0.65
0.95
Table 10: Zero- shot classification results (micro accuracy) on the English MTEB-v2 classification tasks. Bold denotes the best and underlining the second-best score in each column. Best model in each family underlined.
31

---

Published as a conference paper at ICLR 2026
Model
AmzCr
MASS-S
MTOP-D
ToxicConv
IMDB
TweetSE
B77
MASS-I
Acc
Base encoders
bert large uncased
0.49
0.06
0.11
0.51
0.42
0.03
0.03
deberta-x3-large
0.50
0.32
0.03
ModemBBERT-larg
0.50
0.10
0.15
0.50
0.50
0.33
0.04
0.03
0.27
NLJ cross-encoders
bart- large-mnii
0,50
0.55
0.84
0.57
0.93
0.62
0.30
0.43
0.59
roberta
bert-base base-uncassd--il
0.51
0.29
0.62
0.58
0.79
0.63
0.01
0.31
0.47
bert- pott-lrge-unccsed-in
0.57
0.39
0.43
0.55
0.81
0.65
0.08
0.36
0.48
bert-laggeuncasedl nilittrppet
0.60
0.33
0.78
0.07
jeberta-v3-baseenli
0.50
0.50
0.67
0.69
0.91
0.62
0.13
0.34
0.55
deherta large uli
0.51
0.57
0.77
0.73
0.90
0.66
0.38
0.44
0.62
lebhert--3--lrgee-nt purge-nli-triplett
modemneert-base
0.71
0.44
0.47
0.68
0.89
0.61
0.09
0.26
0.52
modernhert-large large-nii
0.57
0.44
0.56
0.69
0.91
0166
0.24
0.38
0.56
modernbert-larger waaeerr--argg-n--ttiiiee
0.91
0.22
Rerankers
ms-marco-Min Mirtoo--MiiiMM------
0.52
0.13
0.23
0.53
0.63
0.41
0.22
0.06
0.34
gre rreereranker--eee modernhert-
bge- see--rranker-barg
0.49
0.54
0.75
0.57
0.70
0.58
0.51
0.53
0.58
hge reranker- large
0.55
0.58
0.82
0.64
0.81
0.61
0.58
0.57
0.64
0.71
0.88
0.66
Qwen3-Rernt Rerunker-8B
0.67
0,80
0.82
0.74
0,95
0.65
0.72
0.74
0.76
Embedding models
all-MiniLM-16-v2
0.51
0.40
c5-base-v2
0.56
0.63
0.83
0.65
0.90
0.59
0.65
0.58
0.67
c5-largo-v2
0.63
0.66
0.81
0.61
0.93
0.59
0.59
0.60
0.68
0.91
bge nge-base-en-vl.. 1.5
0.56
0.64
0.83
0.64
0.90
0.61
0.67
0.64
0.69
hge lge-large-en- large-en-vls.
0.62
0.66
0.87
0.69
0.93
0.62
0.70
0.62
0.72
ge--base-en-v1.5
0.85
0.68
gte-large-en--l.55
0.59
0.71
0.86
0.81
0.94
0.66
0.65
0.66
0.74
gie-modernberi-base
0,54
0.69
0.88
0.74
0.91
0.67
0.66
0.69
0.72
Swen3-EEnbedding-O..
0.88
-nen3--Enbedding-8
0.54
0.78
0.92
0.73
0,95
0.53
0.74
0.72
0.74
Instruction-tuned LLMss
gemma-3-270m-it
0.49
0.07
0.08
0.50
0.52
0.31
0.01
0.01
0.25
gemma-3-lb-it
0.60
0.43
0.15
-lama-3.2-3B-Instren
0.51
0.58
0.58
0.52
0.53
0.44
0.44
0.40
0.50
Qwen3-4B
0.85
0.69
0.80
0.56
0.92
0.65
0.42
0.43
0.67
Thi-4-mini-instruct
0.60
0.45
Qwen3-8B
0.79
0.69
0.83
0.56
0.94
0.66
0.43
0.43
0.67
Mistral-Nemo- Seriul-Nemoo--nsstrree Instruct-2407
0.71
0.65
0.77
0.58
0.95
0.64
0.40
0.40
0.64
Table 11: Zero-shot classification results (macro-averaged recall) on the 8 MTEB (English, v2)
classification datasets and their average (Avg Recall). Bold denotes the best and underlining the second-best score in each column. Best model in each family is underlined.
32

---

Published as a conference paper at ICLR 2026
Model
Amazon C.
Massive S.
MТOP
Toxtc
IMDB
Tweet S.
Banking77
Massive I.
Avg Prec
Base encoders
bett-larcounncsed
0.49
0.04
0.11
0.60
0.55
0.46
0.02
0.00
0.28
deberta--3-Be I--aage
0.51
0.08
0.07
0.51
0.24
0.32
0.02
0.00
0.22
ModemBERI-Harge
0.08
0.22
0.26
0.11
0.05
0.04
NLI cross-encuders
large-miii
ali roberta base
0.53
0.44
0.46
0.66
0.83
0.61
0.13
0.47
0.52
bert-base-uncaane bus---uccccccdniii
bert large uncased-mi nli
0.55
0.61
0.65
0.65
0.84
0.66
0.17
0.53
0.58
bert-large-nncasce -iitipet
debertaiv base nli
0.09
0.62
0.79
0.69
0.91
0.65
0.23
0.49
0.56
deberta-v3-large -alarge-ll
deherta-v3 Saarra----aarg-n----e-
0.59
0,65
0.64
0.72
0,93
0.68
0.35
0.55
0.64
modernbert- base-nli
madernbert-lago--nii
0.55
0.64
0.67
0.69
0,92
0.68
0.29
0.47
0.61
moderbbert-aagee hargee----ag-----rplleee
Rerankers
mu marco MiniL
0.21
0.50
0.53
0.64
0.48
0.44
0.14
0.56
0.59
0.72
0.65
0.86
0.64
0.68
0.58
0.66
bge reranker base
0.50
0.44
0.74
0.57
0.70
0.56
0.57
hge-reranker-lagge
0.53
0.50
0.82
0.65
0.81
0.58
0.66
0.44
0.63
0.82
0.88
0.67
0.66
Qwen3- Rerankeee
0.62
0.79
0.84
0.74
0.95
0.65
0.74
0.71
0.76
Embedding models
Mll-MiniLM--.--e
0.09
0.55
0.69
0.48
0.74
0.45
0.56
0.42
0.50
c5-basc-v2
0.56
0.59
0.82
0.68
0.90
0.59
0.65
0.50
0.66
e5-large-v2
0.58
0.61
0.82
0.93
0.57
0.70
0.55
0.68
cS-mistral-7b-insttee
0.57
0.72
0.87
0.71
0.92
0.64
0.70
0.65
0.72
hge-hase-en-vl1,
0.54
0.60
0.83
0.71
0.90
0.59
0.68
0.54
0.67
hge large en v1.5
0.60
0.62
0.87
0.69
0.93
0.61
0.74
0.54
0.70
gle-hase-en-vi.5
0.61
0.64
0.89
0.88
0.63
0.68
0.54
gle large en-v1.5
0.59
0.66
0.86
0.81
0.94
0.67
0.68
0.59
0.73
ne--modeemberthhase
0.53
0.60
0.86
0.74
0.91
0.66
0.69
0.62
0.70
Qwen]-Emmeele Embedding 0.6B
0.57
0.59
0.85
0.71
0.89
0.60
0.68
0.65
Embeed--nnddiing-- 8B
0.59
0.72
0.90
0.76
0.95
0.74
0.76
0.65
0.76
Instraction-tuned LLMs
semma-3-270m-it
0.04
0.53
0.21
Comma-1-llb-..
0.53
0.25
0.40
0.54
0.67
0.41
0.18
0.20
0.40
3.2.3B
Qwen1-4B
0.78
0.67
0.80
0.73
0.93
0.67
0.74
0.46
0.69
Phi mini
Qwen1-8B SB
0.67
0.68
0.83
0.73
0.94
0.70
0.52
0.48
0.69
Mistral Instruct
precision) BTZSC for the eight MTEB (EN, v2) classification datasets. We report per-dataset precision and overall average precision (Avg Prec). Bold denotes the best and underlining the second-best score in each column. Best model in each family is underlined.
Table 12: Zero-shot classification results
33


Reviews:
[{"review_id": "VJTwlFbEgn", "reviewer": "Reviewer_9fKe", "summary": "The paper proposes a comprehensive benchmark for Zero-Shot Text Classification including 22 datasets, with a focus on comparing three types of approaches: NLI, sentence transformers, and re-rankers. The proposed benchmark is designed to cover task diversity, class granularity, diversity of text domain and length. The paper proposes a primary metric, Micro F1, and the use of NLI performance as a proxy for zero-shot generalization capabilities. Empirical results indicate reranker models achieve the highest overall accuracy, while strong embedding models offer the most favorable balance between speed and accuracy.", "questions": "1. How are the datasets aggregated? Are they not directly aggregated or the labels are aggregated by domain?\n2. I wonder how do generative models perform on such tasks, and would a reasonably sized generative model effectively solve these tasks? What distinct value does this benchmark provide relative to generative models?\n3. Given this dataset, what are some challenges in existing methods and promising directions for future work?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I liked the proposal of a comprehensive evaluation dataset and the clear explanations of the three zero-shot approaches: NLI, sentence transformers, and re-rankers. The evaluation is systematic and includes many model variants. I also found the analysis of scaling across model sizes informative.", "weaknesses": "1. Given this is a dataset/benchmark paper, I'm surprised by how little detail it offers about the construction of the dataset and evaluation procedure. I am having a hard time connecting the underlying datasets and how the final BTZSC evaluation procedure works.\n2. The paper primarily focus on encoder-based architectures, I wonder how do generative models perform on such tasks, and would a reasonably sized generative model effectively solve these tasks? What distinct value does this benchmark provide relative to generative models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RwNe2cFIlt", "reviewer": "Reviewer_yjwX", "summary": "The paper proposes BTZSC, a benchmark for true zero‑shot text classification that compares three families: NLI‑tuned cross‑encoders, embedding models, and rerankers; over 22 English datasets spanning topic, sentiment, intent, and emotion. The primary metric is macro‑F1 (with micro accuracy also reported). The study evaluates 31 models (public and custom), analyzes scaling and the NLI to ZSC transfer, and reports that: (i) rerankers, notably Qwen3‑Reranker‑8B, achieve the highest overall performance (macro‑F1 = 0.72), (ii) strong embeddings (e.g., GTE‑large‑en‑v1.5) “close the gap” and tend to offer the best accuracy–latency trade‑off, and (iii) NLI cross‑encoders are plateauing while scaling primarily benefits rerankers. See Abstract; dataset suite and selection in §3, Table 1 (p. 4), Figure 1 (p. 5); model families in §3.2 (pp. 5–6) and Table 4 (p. 15); results in Table 2 (p. 9); and analyses in Figure 2 (p. 8) and Figure 3 (p. 8).", "questions": "Aren't the used reranking models cross-encoders?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Representative, diverse suite. 22 datasets across four task families; Table 1 and Figure 1 quantify class cardinalities, lengths, and lexical overlap (pp. 4–5).   \n- Clean zero‑shot protocol across families. Unified scoring: NLI log‑odds entailment; embedding cosine; reranker yes/no token probability with a fixed template (§4, Appendix C.3).  \n- Clear headline results. Qwen3‑Reranker‑8B tops macro‑F1 (0.72); GTE‑large‑en‑v1.5 is competitively high among embeddings; Figure 2(b) shows reranker‑favoring scaling; Figure 3(a) visualizes the speed–accuracy frontier (p. 8; p. 9).  \n- NLI to ZSC analysis. Figure 3(b) offers a useful lens on transfer for cross‑encoders and contrasts to embeddings/rerankers (p. 8).", "weaknesses": "1. No zero‑shot LLM‑as‑classifier baseline. The Introduction acknowledges the approach but omits it from evaluation; 8B–32B LLMs are feasible on the stated hardware and would complete Figure 3(a) (pp. 2, 8, 10).\n2. Embedding template opacity. §4 lacks per‑model instruction/template details for E5/GTE/BGE/Qwen‑Embedding; performance is often template‑sensitive (p. 6). These models can be instructed.\n3. Verbalizer multiplicity untested. Single label verbalization per class; no template‑averaging/paraphrase robustness (p. 6; Appendix A notes reusing Laurer et al. 2023 verbalizers). Authors can consider zero-shot augmentations on verbalizers, with a L2-normalization of the average embedding of augmentations like CLIP, or a k-NN approach where majority voting across different augmented forms of verbalizers is used.\n4. Minor inconsistencies. banking77 listed as 77 classes in text (p. 3) vs 72 in Table 1 (p. 4); and the “second‑highest overall 0.64” prose refers to accuracy after defining macro‑F1 as primary (pp. 4, 7, 9).\n5. For completion, it would be interesting to verify whether the conclusions of this paper are reproducible using the same models and evaluation methodology on the classification tasks of MTEB instead. However, such an analysis is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eYpEcX6DzR", "reviewer": "Reviewer_Muat", "summary": "This paper introduces BTZSC, a benchmark comprising 22 datasets to evaluate zero-shot text classification performance across three major model families: NLI-based cross-encoders, embedding models, and rerankers. The authors conduct a broad empirical comparison across 31 models and report that rerankers achieve the highest accuracy, while embedding models offer the best efficiency-performance trade-off.", "questions": "Please refer to weaknesses regarding the lack of technical contributions and analytical depth.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper systematically covers a wide range of datasets and model types, making it one of the most comprehensive empirical studies on zero-shot text classification to date. The release of the BTZSC benchmark and codebase supports reproducibility and could serve as a useful evaluation tool for the community.", "weaknesses": "1. The work lacks technical novelty, where no new algorithm, model, or training strategy is proposed.\n2. The core contribution lies in benchmark, which is largely incremental given the existence of MTEB and similar efforts. The experimental insights are mostly intuitive or expected (e.g., rerankers scale better, embedding models are efficient), and analysis is superficial.\n3. The evaluation design largely mirrors existing work, such as label verbalization and metric selection, with limited innovation in methodology.\n4. The discussion around trade-offs and scaling is descriptive but lacks theoretical depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AeYOU5HHq3", "reviewer": "Reviewer_tZ9R", "summary": "This paper introduces BTZSC, a comprehensive benchmark for evaluating zero-shot text classification performance across three major model paradigms: NLI-based cross-encoders, embedding models, and rerankers. BTZSC unifies 22 English datasets covering sentiment, topic, intent, and emotion classification under a text-label semantic matching framework, where each label is verbalized as a natural-language sentence. Through systematic experiments with 31 models, the paper finds that rerankers achieve the highest zero-shot accuracy, while strong embedding models offer better efficiency-performance trade-offs. The benchmark further analyzes scaling trends, NLI transferability, and latency-accuracy relationships, providing new insights into how different architectures generalize under true zero-shot conditions.", "questions": "1) Is the full benchmark publicly released or planned to be open-sourced? If not, could you clarify the reasons or timeline for public availability?\n2) How do you ensure that the datasets used were not partially included in the pre-training corpora of the evaluated models (such as BERT, DeBERTa, GTE, E5, or Qwen)?\n3) Why did you choose to rely solely on macro-averaged F1, and did you evaluate whether other metrics lead to consistent rankings across models?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) The benchmark systematically evaluates three major paradigms for zero-shot text classification under a unified evaluation protocol. This breadth of comparison provides valuable insight into how different architectures trade off accuracy, scalability, and cost.\n2) The benchmark integrates 22 publicly available datasets spanning multiple task types, which enables a relatively broad assessment of zero-shot performance across heterogeneous domains.", "weaknesses": "1) The paper claims to be \"true zero-shot\", but the 22 datasets used (such as AGNews, IMDb, AmazonPolarity, etc.) are all from public corpora that may have been seen during model pre-training. This means that the model may have indirectly learned the task distribution or category semantics, and strictly speaking, it is impossible to verify the model's migration ability under \"completely new tasks\".\n2) The benchmark relies solely on macro-averaged F1 as the evaluation metric. While this simplifies cross-dataset comparison, it introduces several methodological limitations. F1 alone cannot reveal important trade-offs between precision and recall, nor can it account for task difficulty or class imbalance. Moreover, reranker models are evaluated only on top-1 predictions, ignoring their ranking quality. The absence of per-class variance and statistical significance testing further limits interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)