Title: BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, and Rerankers
Primary area: datasets and benchmarks
Keywords: zero-shot classification, cross-encoder, embedding models, reranker, benchmark, classification tasks, NLPzero-shot classification, cross-encoder, embedding models, reranker, benchmark, classification tasks, NLP

Abstract: Zero-shot text classification (ZSC) offers the promise of eliminating costly task-specific annotation by matching texts directly to human-readable label descriptions. While early approaches have predominantly relied on cross-encoder models fine-tuned for natural language inference (NLI), recent advances in text-embedding models and rerankers have challenged the dominance of NLI-based architectures. Existing evaluations, such as MTEB, often probe embedding models with supervised classifiers atop frozen embeddings, leaving true zero-shot capabilities underexplored. To address this, we introduce BTZSC, a comprehensive benchmark of $22$ public datasets spanning sentiment, topic, intent, and emotion classification, capturing diverse domains, class cardinalities, and document lengths. Leveraging BTZSC, we conduct a systematic comparison across three major model families, NLI cross-encoders, embedding models, and rerankers, encompassing $31$ public and custom checkpoints. Our results show that: (i) modern rerankers, exemplified by Qwen3-Reranker-8B, set a new state-of-the-art with macro $F_{1} = 0.72$; (ii) strong embedding models such as GTE-large-en-v1.5 substantially close the accuracy gap while offering the best trade-off between accuracy and latency; (iii) NLI cross-encoders plateau even as backbone size increases; and (iv) scaling primarily benefits rerankers over embedding models. BTZSC and accompanying evaluation code are publicly released to support fair and reproducible progress in zero-shot text understanding.

TL;DR: A comprehensive benchmark evaluating zero-shot classification performance across cross-encoders, bi-encoders, and rerankers on 25 tasks.