Zero-shot Topical Text Classification with LLMs - an Experimental Study
Authors: Shai Gretz
Alon Halfon
Ilya Shnayderman
Orith Toledo-Ronen
Artem Spector
Lena Dankin
Yannis Katsis
Ofir Arviv
Yoav Katz
Noam Slonim
Liat Ein-Dor
Venue: EMNLP 2023 Findings (2023)
Keywords: topic classification
zero-shot classification
text classification
LLMs

Abstract: Topical Text Classification (TTC) is an ancient, yet timely research area in natural language processing, with many practical applications. The recent dramatic advancements in large LMs raise the question of how well these models can perform in this task in a zero-shot scenario.
Here, we share a first comprehensive study, comparing the zero-shot performance of a variety of LMs over TTC23, a large benchmark collection of 23 publicly available TTC datasets, covering a wide range of domains and styles. In addition, we leverage this new TTC benchmark to create LMs that are specialized in TTC, by fine-tuning these LMs over a subset of the datasets and evaluating their performance over the remaining, held-out datasets.  We show that the TTC-specialized LMs obtain the top performance on our benchmark, by a significant margin. Our code and model are made available for the community. We hope that the results presented in this work will serve as a useful guide for practitioners interested in topical text classification.

Zero-shot Topical Text Classification with LLMs an Experimental Study
Shai Gretz; Alon Halfon; Ilya Shnayderman, Orith Toledo-Ronen, Artem Spector. Lena Dankin, Yannis Katsis, Ofir Arviv, Yoav Katz, Noam Slonim, Liat Ein-Dor
IBM Research
lavishaig.alonnal.lyyaal www.th..areeems...dddaaatttaaaae [yammis.katsis,fft..A.vi]] @ibmm.cm
@il.ibm.com,
Abstract
classification solutions in practice. Hence, gaining better understanding of the current performance of LMs this zero-shot scenario a timely issuc with great practical importance.
Topical Text Classification (TTC) is an ancient, yet timely research area in natural language pro- cessing, with many practical applications. The recent dramatic advancements in large LMs raise the question of how well these models can perform in this task in a zero-shot sce- nario. Here, we share first comprehensive study, comparing the zero-shot performance of a variety of LMs over TTC large bench- mark collection of 23 publicly available TTC datasets, covering a wide range of domains and styles. In addition, benchmark to create LMs that are specialized in TTC, by fine-tuning these LMs over subset of the datasets and evaluating their performance over the remaining, held-out datasets. We show that the TTC-specialized LMs obtain the top performance our benchmark, by signifi- cant margin. Our code and model are made available for the community.' We hope that the results presented in this work will serve as a useful guide for practitioners interested in topical text classification.
Text classification is relatively broad umbrella term, covering tasks such (i) Sentiment Anal- ysis (SA) in various forms (Liu, 2015); (ii) Style Detection e.g., spam filtering (Cormack, 2008), authorship attribution, etc.; and (iii) Topical Text Classification (TTC), where the goal is to associate a text example with one or more topics out of a pre-defined set of topics, or categories (e.g., Lang (1995)).²
leverage this new TTC
The zero-shot scenario is of crucial importance for the latter, TTC task. Specifically, for tasks such as SA. the categories are typically the same between different use cases - Positive, Negative, and Neutral. Hence, in principle, one can rely on existing SA labeled datasets to fine-tune the LM. In contrast, in TTC the list of categories is likely to change from one downstream task to another, hence relying on previously labeled datasets is not straightforward.
1 Introduction
Thus, the main motivation for the present work relies on three intertwined pillars - (i) the special importance of the zero-shot setup for TTC; (ii) the wide range of practical TTC use cases e.g., in do- mains such customer-care, healthcare, and legal (Chalkidis et al., 2022); and (iii) the assumed poten- tial of LMs obtain strong zero-shot performance in TTC tasks.
The recent emergence of Transformer- based Lan- guage Models (LMs) has led significant break- throughs various NLP tasks (Brown al., 2020; Chung et al., 2022). In particular, LMs have shown dramatic performance improvements in text clas- sification (e.g., Zhang et al. (2023)), which one of the most common use cases considered by NLP practitioners. Notably, these improvements are also present in the challenging setting of zero-shot clas- sification, where no labeled data available for the target categories (Yin al., 2019). Manual collection of labeled data is known to be notori- ously costly, complicated, and time consuming, representing a major blocker in the adoption of text
Correspondingly, here we report the results of a comparative study to the best of our knowledge, first of its kind focused performance of plethora of LMs over large col- lection of TTC datasets. Specifically, we consider 23 datasets, across variety of domains, styles and complexity levels, including news, legal contracts. chats and more.
the zero-shot TTC
"These authors equally contributed to this work. 'https: //gi com/IBM/ sero-shot-topicoe cetppiiclleett--eeeeeee text- classification
"We use the terms topics and categorics interchangeably throughout the paper.

---

Moreover, while previous work used LMs for zero-shot TTC without specific tuning for this task, here, we suggest that specializing LMs in the TTC task can provide significant performance gains. To that end, we show that one can fine-tune a LM over existing TTC labeled datasets to significantly improve its performance over new TTC tasks, with categories never seen before. We show the value of this approach in zero-shot TTC by training the LM on a subset of the datasets, and evaluating its performance on the remaining, held-out datasets, To summarize, the main contribution of this pa- per is three-fold: (1) We introduce T7C23, a het- erogeneous collection of 23 TTC datasets, that we propose as a new benchmark for this task; (2) we share the results of a comprehensive comparative study of the zero-shot performance of various LMs at different sizes over T addressing both pre- diction performance and run-time considerations; and finally, (3) we show that TTC23 can be lever- aged to create models with significantly enhanced zero-shot performance in new TTC tasks.
question answering problem, and used the GPT-2 model.
A recently introduced approach for creating zero- shot learners is instruction ttuning (Sanh al., 2021; Chung et al., 2022; Longpre et al., 2023) in which zero-shot gencralization is enabled by map- ping natural language tasks into numan-readabee instructions and finetuning pretrained models with multitask mixture of datasets covering wide variety of tasks. A prominent example for this ap- proach is Flan-T5, which was trained over 1800 tasks, including at least 4 datasets that are topical in nature. While the works mentioned above aimed to create a versatile zero-shot model that can per- form well over a wide range of downstream tasks, our aim here is to explore whether honing-in on a narrower scope of downstream tasks, namely topi- cal text classification, can enable us to improve the zero-shot performance on this type of task only.
A previous paper that focused specifically on benchmarking TTC is the work of Schopf et al (2023), which conducted an evaluation of zero-shot TTC by comparing similarity-basee and NLI-based models on 4 TTC datasets. In our work we expand the set of benchmark datasets and evaluated models, and further suggest to improve upon them via task- specific fine-tuning.
2 Related Work
2.1 Zero-shot Topical Text Classification
Benchmarking TTC was highlighted in Zhang et al. (2015). introducing large-scale datasets, such as AG News, DBPedia and Yahoo Answers.
Zero-shot TTC has been mostly studied as part of wider scope of zero-shot classification tasks in NLP. Yin et al. (2019) have shown the useful- ness of leveraging LMs fine-tuned on NLI datasets for the purpose of zero-shot text classification in general, where topical text classification datasets were only one type of the evaluated tasks. Halder et al. (2020) introduced an approach named TARS, which unifies text classification tasks to pairwise format, and is able transfer knowledge from one dataset to another by leveraging the semantic re- lation between the text and the label in a zero or few-shot setting. Zhong et al. (2021) map text clas- sification tasks into a pairwise dnestion-answering format, where each class is given as a prompt in question format, with Yes/No labels. They also propose the concept of "meta-tuning" suggesting that teaching a model how to solve different tasks in unified format can help it to better generalize unseen tasks. Puri and Catanzaro (2019) pro- posed using generative models for zero-shot text classification. They formulated the text classifi- cation tasks, including TTC, as a multiple-choice
2.2 Evaluation of large LMs
There is a recent rise in attention to evaluation of large LMs in several contexts. Kocoń et al. (2023) and Chen et al. (2023) evaluated ChatGPT and GPT-3.5, respectively, on range of NLP tasks, e.g., sentiment analysis, emotion recognition, and question answering, while Chalkidis (2023) did for ChatGPT specifically on datasets in the legal domain. Zhang et al. (2023) evaluated Flan-UL2, TT-X-,, GPT3.5, and ChatGPT on tasks as- sociated with sentiment analysis, finding that Flan- UL2 achieves comparable or even better results than GPT-3.5 or ChatGPT, despite being magni- tudes smaller. Wadhwa et al. (2023) evaluated GPT-3 and Flan-T5-Large on relation extraction in a few-shot and supervised setting, Parikh et al. (2023) evaluated Flan-T5-XXL and GPT-3 on zero- shot intent detection on benchmark datasets, find- ing that they achieve comparable results.

---

3 Approaches to Zero-shot Topical Text Classification
tasks described via instructions. The instructions are fed to the model as input prompts.
3.1 Existing Approaches
3.2 Our Approach
There are several approaches for dealing with zero- shot TTC. Here we briefty describe a collection of popular methods, highlighting their commonalities and differences. What all the approaches have in common is the key idea of casting different tasks onto single meta-problem. Using this casting, model that was trained solve the meta-problem can also be used to solve other, unseen tasks. The three main meta-problems that have been suggested in the context of text classification are: Natural Language Inference (NLI), Question Answering (QA), and Instruction Tuning. Figure shows how a TTC example is mapped onto each meta-problem.
Most available solutions for zero-shot TTC are based on models that were designed to solve a wider range of tasks. We propose to create models specialized at TTC by leveraging the large amount of publicly available topical datasets. A large col- lection of such datasets from various domains and styles is used to create rich and diverse meta- dataset for TTC, that can be used to fine-tune exist- ing zero-shot models, adapting them to the specific task of TTC.
The details on how we process the datasets and fine-tune different types of zero-shot models on them are presented next, in Sections 4 and 5.
Natural Language Inference (NLI), In this ap- proach, initially proposed by Yin et al. (2019), pre- trained NLI encoder-only models are used as zero- shot text classifiers. These models are pre-trained using large manually labeled datasets for textual en- tailment, such as the MNLI dataset (Williams et al., 2018). The method operates by treating the text to be classified as the NLI premise and constructing a hypothesis for each candidate topic. The hypothe- sis template depends on the target task, where for TTC, the template is usually of the form "this text is about <topic>". The entailment and contradic- tion probabilities predicted by the NLI model are converted to label probabilities for this topic.
The TTC Benchmark
Our aim was to gather a diverse set of datasets containing multiple domains and styles that could serve as a training set for fine-tuning models, and also as a TTC evaluation benchmark. We col- lected datasets for TTC by searching on hugging- face, kaggle, as well as in related papers, focus- ing on datasets where the input is between one and few sentences, in English, and labeled in multi-class or multi-label setting. The collec- tion covers, among others, the following domains and styles: legal both formal (e.g., LEDGAR) and informal (e.g.. Legal Advice Reddit); finance - both from the news (Reuters) and tweets (Finan- cial Tweets); News items - both headlines (News Category Classification Headline) and discussion threads (20 Newsgroups); and Chatbot queries in banking (Banking77) and multi-domain (Massive). Overall we collected 23 datasets, presented in Table I.
Question-Answeriin (QA). In this approach, different tasks are translated into yes/no questions. Zhong al. (2021) created large question an- swering meta-dataset from a collection of manually annotated datasets of different NLP classification tasks, all framed as question answering problems. Next, they used these data to train a binary model for yes/no question answering.
4.1 Converting Datasets from Different Tasks
Instruction Tuning. Another approach for deal- ing with zero-shot text classification is by treat- ing the classification task as a sequence generation task, where the target task is phrased as an input prompt and the text generated by the model is the predicted label. A well-known approach for im- proving the zero-shot performance of generative models is instruction tuning. This approach lever- ages the intuition that NI.P tasks can be described via natural language instructions. Following this intuition, models are fine-tuned on a collection of
Our collection primarily includes datasets associ- ated with detecting the main theme or topic in a given text. To enrich the diversity of the collec- tion, we also considered datasets which were orig- inally targeted at a separate task. For Banking77, Clinc150, and Massive, originally curated for the task of intent detection, took the intent the gold label; for Argument Topic and Clain Stance Topic, which contain arguments associated with de- batable topics, we took the debatable topic as the gold label; and for Contract NLI, which was orig-

---

Text
Label
Class Options
Cleveland Indians pullee within une guese the AL Central Lead by besting the Minnesota Twins...
Sports
Sports/ Technology
Hypothesis Class) This text sbout Sports. Hypothesis(Other Classes)
Label(True Class)
Premise
METATLMENT
Class Options
NLI
The Cloveland...
Lobhe Classes)
ENTAILMENT ENTALLPERT/COMMRODITTI CONTRMIICTION
This text 18 about Technology
CONTRADICTIOON
(uestion[True Class)
Label(True Class)
this about Sports?inThee Cleveland...
YES
Class Options
QA
Question Classes)
Label (Other Classes)
Is this text about Techmology"intte Cleveland...
NO
Instruction
Label
Class Options
Instruction Tuning
Classify the text into one of the catogories: sports, Teshnoloyy text: The cleveland... category:
Sports
Sports/ ww/Tec//oooggg
Figure 1: Framing topical text classification each of the meta-problems representing the common zero-shot approaches.
Dataset
Size
Classes
Multi-label
Domain
20 Newsgroups (l.ang, 1995)
19.0k
20
General
AG News (Zhang et al., 2013)
142. 6k
4
No
General
Argument Topic (Gretz al., 2020)
10.4k
71
No
General
Ba (Casanueva al., 2020)
14.5k
77
No
Finance/Bankinn
Claim Stance Topic (Bar Haim al., 2017)
2.6k
55
No
General
Cline150 (Larson al., 2019)
25.5k
No
General
Contract NLT (Koreeda
Manning, 2021)
7.5k
17
Legal
CUAD (Hendrycks al., 2021)
8.8k
37
Yes
Legal
DBPedia (Zhang et al., 2015)
650k
14
No
Legal
Eli5
(Gao et al., 2021)
91.6k
10
No
General
Financial Tweets (Jia, 2022)
23.6k
20
No
Finance/Banking
HcadQA (Vilares and Gömez-Rodriguzz, 2019)
8k
No
Medical
Law Stack Exchange (Li et 2022)
2.8k
16
Legal
LEDGAR (Chalkidis al., 2022)
88.5k
99
No
Legal
Legal Advice Reddit (Li et al., 2022)
108,8k
11
No
Legal
Massive (FitzGerald et al., 2022; Bastianelli et al., 2020)
18.50
60
No
General
Medical Abstracts (Schopf al., 2023)
15.5k
No
Medical
News Category Classification Headline
229.5k
40
No
General
(Misra and Grover, 2021; Misra, 2022)
Reuters (Apté al., 1994)
1k
20
Yes
Finance/ Braance/Banking
Sentivent (Jacobs, Gilles and Hoste, Veronique, 2022)
7.5k
18
Yes
Finance/ Bhnance/Banking
Unlair ToS (Chalkidis et al., 2022)
11.6k
8
Yes
Legal
Xglue (Liang ct al., 2020)
130.0k
10
No
General
Yahoo Answers (Yin et al., 2019)
145.4k
10
No
General
Table 1: List of datasets the 27
benchmark collection.
inally curated for identifying entailment between premises and hypotheses in contracts, we took the category of the hypothesis as the gold label.
contains zero or more categories (depending on the dataset), For datasets that originally did not contain train-dev-test split, created
4.2
Data Formatting
4.3 Label Cleansing and Rephrasing
To ease data processing, we converted all datasets to a unified format containing a single text col- umn and single label column, where each label
Zero-shot TTC relies on a good semantic represen- tation of the topic. For it to be effective, this repre- sentation should be clear and grammatically sound,

---

and also consistent between different datasets. This led us rephrase the categories in the following ways (see Appendix for the full list of changes):
Out-of-domain. In this split, different folds do not share datasets from the same domain. The motivation for this split to evaluate how well a model performs when tested on a domain not present in training. We determined for each dataset its domain manually, ending up with 4 folds, presented in Table 1.
1. We rephrased categories that were in readable technical format (c.g., in 20 Newsgroups, we rephrased att.ithessm sm to atheism).
5.2.2 Building Unified Dataset
2. We rephrased categories that might not be grammatically correct when prepended with prompt (e.g., Clinc150, we rephrased find_phone to finding a phone).
When fine-tuning model in the leave-one-foldoout setup, create merged train and dev sets. The merged train set is comprised of the train of all datasets in the folds we train on (and likewise, the merged dev set). For the train set, we also sam- ple at most 100 examples from each category to prevent over-dominance of highly populated cate- gories, For evaluation, we consider the test sets of all datasets in the held-out fold separately, and they not sampled.
3. We removed examples associated with cate- gories that do convey semantic informa- tion (e.g., topic general LEDGAR).
4. For the Reuters dataset, many categories were either acronyms or required domain exper- tise to parse. Thus, we kept examples asso- ciated with the top-20 prevalent categories in the dataset, and rephrased these categories to make them more readable.
For building the merged train/dev sets for NLI- based models, we convert the positive examples a pairwise format, where each example contains the text input and the hypothesis template "This text is about <topic>.", separated by [SEP], with an ENTAILMENT label. In case the example contains multiple positive topics (in multi-label setting), create multiple pairwise positive examples, for each positive topic.
5. We performed further cleansing across all datasets to remove underscores and convert to lower case.
5 Experimental Setup
For each positive pairwise example we add a negative pairwise example whose topic is selected at random from the categories of this dataset (ex- cluding the positive topies of this example), with CONTRADICTION label.
Next we describe the setup in which we evaluate the different methods for zero-shot TTC.
5.1 Datasets
We use the 2TC23 benchmark described in Sec- tion 4 containing 23 datasets 18 multi-class and 5 multi-label for training and evaluation.
For instruction-tuned models, we similarly merge the datasets in the training folds. The in- put for each text comprised of the instruction, the candidate categories, and the text, and the out- put is the target category. If the example originally contains more than one category, we choose one randomly. Examples with no categories are ignored in training. We use the instruction in Figure 1.
5.2
Evaluating Fine-tuned Models on 23
5.2.1 Leave-one-fold-ont
To evaluate fine-tuned models in zero-shot setting, we split 2TC23 into I folds and employ leave- one-fold-out setup, where we train on k-1 folds and evaluate on the remaining fold.
5.3 Inference
We consider two fold-splits:
Encoder-only Models. For all encoder-only models run pairwise inference of each ex- ample with all candidate categories.
In-domain. In this split, different folds share datasets from the same domain. The motivation for this split is to evaluate how well model per- forms when tested on similar domain that was present in training. We split to 3 folds containing 8, 8, and 7 datasets, respectively. Unless stated oth- erwise, this fold-split is used for all experiments. The folds are presented in Appendix A.
Encoder- -eccder Models. For instruntion- tuned models, when evaluating multi-class datasets we construct the same instruction that contains the task description and all candidate categories, and ask the model to generate the predicted category.

---

Classification tasks are a special case of gener- ation task where the generated tokens are expected to come from a pre-defined list of categories. Thus, we implemented a logits processor that restricts the calculation of tokens to take into account only the tokens that are permitted according to the candi- date categories. This avoids generating texts which cannot be exactly matched to any category and im- proves the performance. For example, if there two candidate categories, "world news" and "tv and film", and the first generated token is "world", the second token must be "news".
Instruction-tuned models. We evaluate Flan-T5- Largc/XL/XXL. Four of the datasets in TTC were part of Flan's 's training, thus we also report results on a subset of TTC that excludes them. Given that datasets may contain a large set of candidate categories, increasing the size of the prompt, we use a sequence length of 2018. We use a greedy decoding method, since our aim is to generate the tokens representing the most probable class.
We also evaluate models fine-tuned over TTC in the leave-one-fold-our setup described above. Implementation details can be found in Ap- pendix B. We consider the following fine-tuned models:
Multi-label datasets have been relatively less ex- plored in the context of using instruction-tunet models for classification. We propose to frame the classification as a binary classification task. Instead of asking the model to generate single category name out of a pre-defined list of candidates (as the multi-class setting), we present the model with each candidate category separately, along side its negation (e.g.. sports, not sports), and ask the model to choose between the two. The score for each <text, category> pair is the probability given by the model for generating that category name.
NLI-based FT. We fine-tune RoBERTa-large- NLI and DeBERTia-lage-NLL.
Instruction-tuned FT. We fine-tune Flan-T5- XXL. In order to be able to fine-tune this model, we use the efficient fine-tuning ap- proach of Low Rank Adapters (LoRA) (Hu et al., 2021). We fine-tune the model for 3 epochs and use the dev set for early stopping.
5,4.1 Decoder-only Models
5.4 Models
Recent advancements exhibited in GPT-4 and Chat- GPT suggest they could be suitable for TTC as well. However, there are several issues associated with evaluating these models. First, assessing the zero-shot performance of the GPT family is chal- lenging due its undisclosed training data, poten- tially encompassing supervised training from our evaluation datasets. Second, given its paid nature, extensive inference at the scale presented in this work is not only costly but also non-reproducible should the service evolve in the future. Thus, we excluded them from this paper.
We consider the following off-the- models:
S-BERT (Reimers and Gurevych, 2019). In addi- tion to the upproaches in Section 3, we also evaluate S-BERT. We use the all-mpnet-base-v2 model.
QA-based. We the RoBERTa- model released in Zhong et al. (2021).² For each inferred pair, we calculate the probability of the la- bel "Yes". We use a single label description (ques- tion), 'Is this text about <topic>?', which one the manually annotated label descriptions used by Zhong et al. (2021), and matches the TTC task. Three of the 23 datasets were part of the train- ing set of this model, thus also report results on subset of 23 that excludes them.
5.5 Metries
Multi-class. For multi-class datasets, we take the highest scoring predicted category per example, and report macro-averaged fl over all categories.
NLI-based. We al., 2019)t and DeBERTa--lagge-NN (He al., 2021). For each inferred pair, we calculate the probability of the label ENTAILMENT divided by the probability of the label CONTRADICTION.
MargeeRa-larg--oe (Liu
Multi-label. For multi-label datasets, to avoid sensitivity to any decision threshold, we report macro-average AUC-ROC over all categories. We report the average of 3 seeds.
6 Results and Analysis
"https::/huugginffcccccoo// cw/.riiiizzhong/
roberta-lee mattt--tt-tt
We present results averaged on all datasets in Ta- ble and full results in Tables and 5 in the Ap- pendix.
htttps://huggingfaaa
http://ggaaa....oaoiie/a co/Moritalaures
DeBERTa-v3- -arge-e--laaeaaeee
www.laarreel-------eee mwwi- feii-ffeer-nnnii anli- ling-wann wanli

---

Model
Model Type
#Params
MC
MC*
MC**
ML
Off-the-shelf Models
S-BERT
110M
52.66
51.89
54.15
85.49
RoBERTa-Lanee Large-QA
QA-based
355M
54.21
88.12
RoBERTa-Large
NLI-based
355M
51.05
47.89
51.26
82.97
DeBERTa-Large
NLI-based
435M
54.40
51.08
55.13 15
88.26
Flan-T3-Larg
Instruction-tuned
770M
54.89
86.57
Instruction-tuned
3B
61.39
89.47
Man-TT-XXL
Instruction-tuned
11B
64.79
89.72
Models fine-tuned on TTC
RoBERTa-Large-NN FT
NLI-based
355M
58.59
57.51
59.34
90.64
DeBERTa-Large--LI
NLI-based
435M
64.00
63. 14
65.36
93.19
Flan-T5-XXL
Instruction-tuned
11B
67.32
89.56
Table 2: TTC results 18 multi-class and 5 multi-label datasets in TTC23 Top: off-the-shelf zero-shot models. MC stands for multi-class datasets, and ML for multi-lahel datasets. MC are the subset of datasets not included in Flan models training. MC** are the subset 15 datasets not included in the QA model training. For MC/*/** report macro-fl fl and for ML macro-AUC-ROC averaged over all respective datasets.
Bottom: fine-tuned models over TTC23 in leave-one-fold-on
6.1 Off-the-shelf Models
averaged fl score of 67.32. As can be seen in Table 4, it is the best model for 10/14 of the MC* datasets. Interestingly, DeBERTa-Large-NLA FT outperforms the much larger Flan-T5 -S--L,, which further highlights the advantage of the task-specific training.
Considering the top part of Table 2, Flan-T5-XLL model is clearly the best zero-shot model for TTC. As expected, the performance of Flan-based mod- cls increases with model size (see MC* column). Note, that QA and Flan models were trained multiple tasks (including TTC ones), making them a stronger baseline compared to the NLI-based models which were trained on single and some- what different entailment task. It is also worth noting that S-BERT is competitive with off-the- shelf encoder-only models, representing a much faster cost-effective alternative when resources are l
Focusing on encoder-only models, and the MC** column, fine-tuning on TTC yields better perfor- mance compared to the QA-based model, that uses different training scheme.
6.3 Comparison to Leave-one-ommain-out
We compare the results of our in-domain setup to out-of-domain setup, where the evaluated datasets in each fold come from domain not present in the training data, using DeBERTa- LagERRa-Large-NJJ FT, This enables to quantify the impact of domain similarity between train and test on model performance. The results are in Table 3. When moving to an out- of of--dmmann setup macro-f1 and macro-AUC-ROC drop by 1-2 points, as expected. The out-of-domain setup is more challenging, but not necessarily more realistic. Given that envision model trained on hundreds of classes and multiple domains, it might be reasonable to assume that real-worte applications this model will be tested over datasets is already somewhat familiar with.
6.2 Impact of Fine-Tuning on 23
The results of models fine-tuned over 23 in ceave-one-fold-ee setup presented the bot- of Table Remarkably, fine-tuning RoBERTa and DeBERTa models significantly improves their macro-fl performance on MC datasets by and 10 points, respectively; and by 10 and 12 points, respectively, over the MC* datasets. Fine-tuning Flan-T5-XXL yields more moderate, yet signif- icant improvement of 2.5 fl points on the MC* datasets, leading to the top zero-shot performance in MC experiments with an impressive macro-
'I addition to the models presented here, we experimented with the Llama family of models. Our trials in zero-shot clas- sification with them on T7 yielded significantly inferior results, leading to its exclusion.
6.4 Does Category Similarity Help?
As an additional measure for evaluating the impact of training data on model performance, we explore

---

whether high performance on categories in the test fold correlated with the occurrence of semanti- cally similar categories in the train set. To that end, for each category in a test fold we find the score of its most similar category in the train folds us- ing S-BERT. The correlation between these scores and the respective fl scores in the DeBERTa-Large- NLI FT run is insignificant (-0.03). Furthermore, observe relatively
Model
MC
MI
DeBERTa-Larg NLIFT
64L00
93.19
DeBERTa-L.ae Large-NN (OOD)
61.83
92.04
Table 3: TTC over 18 multi-class and multi-label datasets in T7C23, comparing in-domain (top) to out-of- domain (bottom) fold-splits.
and 8.02 Kchar/s respectively. Full analysis is avail- able in Table 6 in the Appendix. Next, share few observations that emerge from this analysis. There is large variance between different datasets, as text length and number of categories im- pact the throughput. For example, throughput on Clinc150 is 0.43 Kchars/s compared to 22.1 Kchars/s on Legal Advice Reddit. Similar variance is seen in other models.
in anecdotal inspection, low performance for categories in the test fold even though the exact same category name was included in the train folds. E.g., the category greeting ap- pears both in C5 and Massive, however the fl results for this category are lower compared to other categories. Thus, overall it seems that the semantic similarity of a category name to another category included in the training data has no signif- icant influence over performance, possibly due to the magnitude and diversity of the training data we consider.
TS-XXL's
For most datasets, the throughput of DeBERTa- Large-NLI is higher, i.e., better than that of Flan- T5-XXL, despite making more inference calls per text. For example, in Xglue the throughput of DeBERTa-Large-NT is more than 4 times greater than with Flan-T5-XXI. The only exception to this is LEDGAR, which contains long texts and a large number of categories. For this dataset, the through- put of Flan-T5-XXL is two times larger. This is presumably because Flan-T5-XXI. processes the long text only once, and concatenates the candidate categories single inference, while DeBERTa- Large-NLI infers the text repcatedly for each can- didate category. Coupled with the results discussed in Section 6.2, this analysis highlights the run-time vs. quality trade-off, as the superior quality of Flan- T5-XXL comes at the cost of longer run-time.?
6.5 Run-time Analysis
When choosing which model to run, a relevant as- pect for a practitioner is run-time. To this end, analyze the run-time of the different models. Each of the models we consider uses different tech- nique for running predictions: S S-BERT calculates the embeddings of the input texts and categories separately and calculates their cosine similarity. Flan-T5-X -5-XI. concatenates all the candidate cat- egories to a single prompt with additional tokens for instruction, and runs inference each in- put text thus, the number of categories directly impacts the prompt length. DeBERTa-Large-NL prepends a short template to each text, and runs inference for each category separately. Thus, the number of categories directly impacts the number of inferences per input text.
For S-BERT, the effect of calculating category embeddings and cosine similarity is negligible, and the model does not use any prompt. Aside from the fact that it is the smallest model, S-BERT naturally has the best throughput across all datasets, at the cost of lower fl.
Inference over large datasets naturally takes more time than inference on smaller datasets. To compare run-time between datasets of different sizes, we measure the throughput using the Kchar/s (kilo characters per second) metric, dividing the total length of the dataset input by the run-time.
7 Conclusions
TTC is an ancient problem in computer science research, dating back to 1961 (Maron, 1961). The recent advancements in large LMs led us to ex- plore how well they perform on this problem in zero-shot setting. In this paper we introduce comprehensive evaluation of existing approaches to zero-shot TTC over a diverse set of datasets. 23 Our results indicate that large LMs indeed
We use an A A100 GPU with 40GB of memory. For each method, we pick the inference batch size that allows minimal run-time as measured in sec- onds. We use fp16 quantization for both DeBERTa- Large-NLI and Flan-T5-XXL.
The average throughput of S-BERT, DeBERTa- Large-NLI, and Flan-T5-XXL. is 118.73, 21.67,
Note, Tlan-T5-XXI also has relatively high hardware constraints, though we did not analyze this aspect in detail.

---

exhibit significant improvement over the smaller encoder-only models that were the main focus on previous studies on zero-shot TTC ge. (Yin et al., 2019; Schopf et al., 2023))-
We call all to join in advocating for the prompt and safe return of the abductees, as we stand gether in the pursuit of justice and peace.
Furthermore, we show that fine-tuning RoBERTa and DeBERTa models as well as Flan-T5-XXL over existing TTC datasets can significantly boost their respective zero-shot performance on new TTC datasets, with category names not included in the training data. In other words, even LLMs that represent very strong baselines, can be further significantly improved with additional fine-tuning focused on the target task. Our run-time analysis has shown that this superior performance comes with cost of additional run-time, something the practitioner needs to take into account.
Limitations
The approach taken in this work has a few limita- tions:
1. For reasons detailed in Section 5.4.1, we do not present an evaluation of decoder-only models such as GPT-4 or ChatGPT.
2. For Flan-based models, it could be that with better prompt engineering, e.g., including try- ing few-shot prompts, one could improve their performance.
Our fine-tuned Flan-T5-XXL model obtained very impressive zero-shot performance macro- averaged fl of 67.32 when averaging across 14 multi-class datasets. Nonetheless, these results typically still not satisfactory in practice, and expect that a traditional process of collecting la- beled data for the target task under consideration to further fine-tune the model would be required. That said, strong zero-shot TTC model can serve as an excellent starting point to perform domain adaptation on target dataset, e.g., by means of self-training (Gera et al., 2022) or active learning (Fin-Dor et al., 2020; Shnarch al., 2022), plan to explore in future work.
3. Our approach for classification of multi-label datasets with the generative models is rough attempt to adapt these models to this setup. It could be that other approaches for multi-label classification could be utilized, e.g., by asking the model to generate zero or more categories.
4. We analyze the potential impact of the training data on model performance in Sections 6.3 and 6.4. However, perhaps there are other, more subtle ways in which the training data impacts the performance which are not di- rectly evaluated in this work.
8 Ethics and Broader Impact
We do not attempt to fine-tune the QA-based model, which could have benefited from as well.
This paper is submitted in the wake of tragic terrorist attack perpetrated by Hamas, which has left our nation profoundly devastated. On October 7, 2023, thousands of Hamas terrorists infiltrated the Israeli border, launching a brutal assault on 22 Israeli villages. They methodically moved from home to home brutally torturing and murdering more than thousand innocent lives, spanning from infants to the elderly. In addition this horrifying loss of life. hundreds of civilians were abducted and taken to Gaza. The families of these abductees have been left in agonizing uncertainty, as no infor- mation, not even the status of their loved ones, has been disclosed by Hamas.
6. We consider datasets containing sentences. paragraphs, or short sections. We do not test our approach document-level inputs, though in theory this should be feasible.
This type of work entails several degrees of freedom what templates to use, which datasets to evaluate on, what tyyer-parammeten to tune, which runtime measures to consider, etc. We attempted to prioritize them in or- der to cover the issues we consider most im- portant. However, dedicating more time and effort to these items could have yielded addi- tional insights.
We fervently call for the immediate release of all those who have been taken hostage and urge the academic community to unite in condemnation of these unspeakable atrocities committed by Hamas.

---

References
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery. Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai. Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin. Adam Roberts, Denny Zhou, Quoe Le, and Jason Wei. 2022. Scaling instruction-finctuned language models.
Chidanand Apté, Fred Damerau, and Sholom M. Weiss, 1994. Automated learning of decision rules for text categorization. ACM Transactions on Information Systems, 12(3):233-254
Roy Bar- Indrajit Bhattacharya, Francesco Din- uzzo, Amrita Saha, and Noam Slonim. 2017. Stance classification of context-dependent claims. In Pro- ceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistics: Volume 1, Long Pupers, pages 251 1, Valencia, Spain. Association for Computational Lin- guistics.
Gordon V. Cormack. 2008. Email spam filtering: A systematic review. Foundations and Trends@) in In- formation Retrieval, 1(4):335-455.
Emanuele Bastianelli, Andrea Vanzo, Pawel Swietojan- ski, and Verena Rieser. 2020. SLURP: A spoken lan- guage understanding resource package. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLPP), pages 1252-72622. Online. Association for Computational Linguistics.
Liat Ein-Dor, Alon Halfon, Ariel Gera, Eyal Shnarch, Lena Dankin, Leshem Choshen, Marina Danilevsky, Ranit Aharonov, Yoav Katz, and Noam Slonim. 2020. Active Learning for BERT: An Empirical Study. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 7949-7962, Online. Association for Computa- tional Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Πlya Sutskever, and Dario Amodei, 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash, Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis, Gokhan Tur, and Prem Natara- jan. 2022. Massive: A Im-example multilin- gual natural language understanding dataset with applogically-diverss languages.
Jingsong Gao. Qingren Zhou, and Rui Qiu. 2021. ELI5-Category: a categorized open- open-domain qa dataset. eli5_categorr.
co/datasets/
Inigo Casanueva, Tadas Temeinas, Daniela Gerz, Matthew Henderson, and Ivan Vulic. 2020. Er- ficient intent detection with dual sentence en- coders. In Proceedings of the 2nd Workshop on NLP for ConvAl ACL 2020. Data avail- able at http:///thlbbbcoooo/bAADAAVsssss datasets.
Ariel Gera, Alon Halfon, Eyal Shnarch, Yotam Perlitz, Liat Ein-Dor, and Noam Slonim. 2022. Zero-shot text classification with self-training, In Proceedings of the 2022 Conference Empirical Methods in Nat- ural Language Processing, pages 1107-1 119, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Ilias Chalkidis. 2023. ChatGPT may pass the bar exam soon, but has a long way go for the LexGLUE benchmark.
Shai Gretz, Roni Friedman, Edo Cohen-Karlik, As- saf Toledo. Dan Lahav, Ranit Aharonov, and Noam Slonim. 2020. A large-scale dataset for argument quality ranking: Construction and analysis. Proceed- ings of the AAAI Conference on Artificial Intelligence. 78005):7805-7813.
Ilass Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, lon Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUT: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. 2020. Task-aware representation of sen- tences for generic text classification. In Proceed- ings of the 28th International Conference on Com- putational Linguistics, pages 3202-3213, Barcelona, Spain (Online). International Committee on Compu- tational Linguistics.
1: Long Papers), pages 43 13-4333,, Dublin, Ireland. Association for Computational Linguistics.
Xuanting Chen, Junjie Ye, Can Zu, Nuo Xu, Rui Zheng. Minlong Peng. Jie Zhou, Tao Gui, Qi Zhang, and Xuanjing Huang. 2023. How robust is GPT-3.5 to predecessors? comprehensive study on language understanding tasks.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced

---

BERT with disentangled attention. In International Conference on Learning Representations.
Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Fenfei Guo, Weizhen Qi. Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, Xiaodong Fan, Ruofei Zhang, Rahul Agrawal, Edward Cui, Sining Wei, Taroon Bharti, Ying Qiao, Jiun-Hung Chen, Winnie Wu, Shuguang Liu, Fan Yang, Daniel Campos, Rangan Majumder, and Ming Zhou. 2020. XGLUE: A new benchmark dataset for cross-lingual pre- pre-training, un- derstanding and generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6008 -0186 Online. Association for Computational Linguistics.
Dan Hendrycks, Collin Burns, Anya Chen, and Spencer Ball. 2021. CUAD: An expert-annotated NI.P dataset for legal contract review. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Truck (Rownd 1).
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09685.
Bing Liu. 2015. Sentiment Analysis: Mining Opinions, Sentiments, and Emotions. Cambridge University Press.
Jacobs, Gilles and Hoste, Veronique. 2022. SEN- TiVENT enabling supervised information extrac- tion of company-specific events in economic and financial news. Language Resources and Evaluation, 56(1):225-257.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer. and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized bert pretraining ap- proach.
Yuheng Jia. 2022. Twitter financial news dataset.
https: / wttp:///ww.....ettttttttteeetttt twitter-financial-nee toppe//eee-ooopicccree/aaaa
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe Le, Barret Zoph. Jason Wei, and Adam Roberts, 2023, The Flan Collection: Designing data and methods for effective instruction tuning.
Jan Kocoñ, Igor Cichecki, Oliwier Kaszyca, Mateusz Kochanek, Dominika Szydło, Joanna Baran, Julita Bielaniewicz, Marcin Gruza, Arkadiusz. Janz, Kamil Kanclerz, Anna Kocoñ, Bartiomiej Koptyra, Wik- Piotr Milkowski, Marcin Oleksy, Maciej Piasecki, Eukasz Radliński, Konrad Wojtasik. Stanistaw Wožniak, and Prze- myslaw Kazienko. 2023. ChaiGPT: Jack of all trades, master none. Information Fusion, 99:101861.
toria
M. E. Maron. 1961, Automatic indexing: An experi- mental inquiry. J. ACM, 8(3):404 417.
Rishabh Misra. 2022. News category dataset. arXiv preprint arXiv:2209.1 14429.
Rishabh Misra and Jigyasa Grover. 2021. Sculpting Data for MI.: The first act of Machine Learning.
Yuta Koreeda and Christopher Manning. 2021. Con- tractNLI: A dataset for document-level natural lan- guage inference for contracts. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1907-1919, Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Soham Parikh, Quaizar Vohra, Prashil Tumbade, and Mitul Tiwari. 2023. Exploring zero and few-shot techniques for intent classification.
Raul Puri and Bryan Catanzaro. 2019. Zero-shot text classification with generative language models. In 3rd Workshop on Meta- Meeaning NeurlPS 2019.
Ken Lang. 1995. NewsWeedrr: Learning filter net-
news. Machine Learning. Proceedings of the Twelfth International Conference on Machine Learn- ing, Tahoe City, California, USA, July 92, 1995, pages 331 231-3399 Morgan Kaufmann.
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert, Sentence embeddings using siamese newt-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguistics.
Stefan Larson, Anish Mahendran, Joseph Peper, Christopher Clarke, Andrew Lec, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach. Michael A. Laurenzano, Lingjia Tang, and Jason Mars. 2019. An evaluation dataset for intent classification and out-of- scope prediction. In Proceedings of the 2019 Confer- ence on Empirical Methods Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint 10.00077
Tim Schopf. Daniel Braun, and Florian Matthes. 2023. Evaluating unsupervised text classification: Zero- shot and similarity-basel approaches.
Jonathan Li. Rohan Bhambhoria, and Xiaodan Zhu. 2022. crrameter-efficient legal domain adaptation. In Proceedings of the Natural Legal Language Pro- cessing Workshop 2022, pages 119-129, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Eyal Shnarch, Alon Halfon, Ariel Gera, Marina Danilevsky, Yannis Katsis, Leshem Choshen, Mar- tin Santillan Cooper, Dina Epelboim, Zheng Zhang,

---

Dakuo Wang, Lucy Yip. Liat Ein-Dor Lena Dankin, Iyya Shnayderman, Ranit Aharonov, Yunyao Li, Naf- tali Liberman, Philip Levin Slesarev, Gwilym New- ton, Shila Ofek-Koifman Noam Slonim, and Yoav Katz. 2022. Label Sleuth: From unlabeled text to a classifier in a few hours. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. As- sociation for Computational Linguistics.
Fold2: Argument Topic, CUAD, DBPedia, News Category Classification Headline, Elis-category, Financial Tweets, Law Stack Exchange, Massive. Fold3: C0i 20 Newsgroups, Contract NLI, LEDGAR, Legal Advice Reddit, Medical Ab- stracts, Xglue, Sentivent.
B Model Implementation
David Vilares and Carlos Rodriguzz. 2019. HEAD- HEAD-QA: A healthcare dataset for complex reason- ing. In Proceedings the 57th Annual Meeting of the Association for Computational Linguistics, pages 960-966, Florence, Italy, Association for Computa- tional Linguisties.
For DeBERTa-based models, we fine-tune for 3 epochs without carly-stopping using a learning- rate of 5e-6, a batch-size of 32, and a maximum sequence length of 256.
For RoBERTa-based models, we use maxi- mum sequence length of 128, which is the limit of the RoBERTa-Large -AA released model. For the RoBLRRT-Large-NLI model, we fine-tune for 3 epochs without early stopping using a learning-rate of le-5, and a batch-size of 32.
Somin Wadhwa, Silvio Amir, and Byron C. Wallace. 2023. Revisiting relation extraction in the era of large language models.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume 1 (Long Papers). pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
When fine-tuning Tlan-T5-XXL with LoRA, we adapt only the (query) and v (value) projections in the transformer, and set rank=16, alpha=32 and dropout=0.05. We set the learning rate to 3e-5, the batch size 16, the gradient accumulation steps to 16, the input sequence length to 512, the warmup steps 2 and use paged Adamw optimizer.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.
C Full MC Results
Tables 4 and 5 present the full results of our evalu- ation over 2TC
D Runtime Results
Table 6 presents the runtime analysis over all multi- class datasets of TTC
Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023. Sentiment analysis the era of large language models: A reality check.
E Rephrasing Labels
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Pro- cessing Systems, volume 28, Curran Associates, Inc.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021. Adapting language models for zero-shot learn- ing by meta-tuning on dataset and prompt collections. In Findings of the Association for Computational Punta
Linguistics: EMNLP 2021, pages Cana, Dominican Republic. Association for Compu- tational Linguistics.
A In-domain Folds
In the in-domain fold-split, these are the folds considered:
Fold1: Reuters, Claim Stance Topic, Unfair ToS, HeadQA, Banking77, AG News, Yahoo Answers.

---

Dataset
RoBERTa-
RoBERTa-
RoBERTa- Large-
DeBERTa-
DeBERTa- Flan-
Flan-
Flan- XXL
Flan-
BERT
Large-
Large-
Large-
Large- NLIFT
Large
XL
XXL
NLI
NLIFT
QA
NLI
FT
20
News-
59.68
50.42
53.53
41.16
56.63
66.87
57.1
62.33
62.28
72.2
group
AG News
59.85
70.63
76.69
68.67
79.55
Argument
95.46
81.75
91.7
76.38
89.38
96.65
88.61
94.19
95.73
97.83
Topic
Banking77
60.42
36.91
56.99
49.96
66.3
55.6
62.76
66.59
68.89
Claim
69.82
42.98
75.75
50.35
56.94
78.75
73.23
75.99
82.21
Stance
Topic
Clinc150
69.15
51.87
71.98
54.1
54.73
79.06
61.4
72.92
79.11
83.47
DBPedia
68.88
82.56
82.54
84.1
92.76
90.18
Fli5-
45.36
48.89
39.47
50.97
53.06
50.64
50.3
49.25
56.34
57.11
category
Financial
30.21
32.32
46.61
40.82
27.94
48.68
46.97
53.82
52
49.48
Tweets
HeadQA
40.94
41.35
30.97
45,83
44.24
36.81
Stack
45.73
54.03
37.54
54.86
57.66
59.91
52.35
52.78
58.17
Exchange
LEDGAR
23.28
33.77
46.41
37.61
18.5
18.90
31.89
14.73
17.73
55.86
L.egal
Ad-
56.78
63.69
66.69
68.76
67.45
72.21
62.56
67.94
75.59
77.99
vice Reddit
Massive
50.61
48.78
58.31
50.9
54.22
66.61
46.22
61.27
64.82
68.54
Medical
58.4
53.39
53.74
54.34
53.6
59.13
54.04
62.48
63.71
60.69
Abstracts
News
24.06
25.5
28.57
24.77
30.53
29.34
33.04
37.64
36.91
Category
Classi-
fication
Headline
Xglue
37.56
43.12
54.82
54.97
50.24 58,42
59.58
58.83
65.67
65.13
63.25
Yahoo An-
51.64
54.02
59.23
61.59
swers
Table Full results of all models on 18 MC TTC datasets.
Dataset
S-
RoBERTa- RoBERTa- Large- NLI
RoBERTa- Large- QA
DeBERTa- Large- NLI
DeBERTa Large- NLI
Flan-
Flan-
Flan- T5-
Flan-
BERT
Large- NLIFT
T5-
T5-
T5-
Large
XL
XXL
XXL
FT
Contract
75.55
71.8
80.33
76.31
74.95
87.10
75.85
76.66
75.42
76.17
NLI
CUAD
88.49
86.04
94.65
92.10
88.96
96.71
92.27
93.66
92.46
92.39
Reuters
93.58
97.35
97.26
96.12
98.31
87.83
93.62
94.7
Sentivent
78.75
80.75
83.79
86.18
85.52
85.71
81.88
85.32
87.17
86.29
Unlair ToS
91.06
88.05
97.08
88.73
95.76
98,10
95.04
98.08
98.36
98.23
Table Full results of all models on 5 ML TTC23 datasets.

---

Dataset
# Categories
Average Text Length
S-BERT
Deberta- Deberta-Laage-NIII
Flan-T5-XXI.
20 Newsgroups
20
1925.64
274.96
21.94
15.19
AG News
4
232.49
87.12
54.33
13.78
Argument Topic
71
116.73
64.91
2.97
1.72
Banking77
59.04
29.16
1.45
0.85
Claim
Stance
79.47
45.13
2.68
Topic
Clinc150
150
38.02
25.82
0.57
0.43
DRPedia
14
280.08
104,60
17.60
5.87
Eli5-category
10
82.88
44,21
15.21
4.30
Financial
135.74
59.96
8.15
3.10
Tweets
HcadQA
116.80
39,22
28.81
4.05
Stack Ex-
849.40
129.22
20.82
18.33
change
LEDGAR
715.78
117.31
3.91
8.37
Legal
Advice
1292.77
202,30
37.16
22.51
Reddit
Massive
60
31.36
22.25
1.24
0.68
Medical
Ab-
5
1185.45
182.09
72.57
17.66
stracts
News Category
59.09
38.30
3.15
1.39
Classification
Headline
Xglue
3971.01 508.63
589.28
71.59
16.42
Yahoo Answers
10
83.47
26.05
8.84
Table 6: Throughput in Kchars/sec Tesla A100 with 40Gb.

---

dataset
class name
rephrased class name
rephrased type
claim stance topic
the one-child policy
the one-child policy of the republic of china
lower case
of the republic of China
claim stance topic
make physical edu-
physical education
motion to topic
cation compulsory
claim stance topic
subsidize the grow-
the growing of tobacco
motion to topic
ing of tobacco
claim stance topic
American Jobs Act
american jobs
lower topic
claim stanee topic
nations have
nations right nuclear weapons
right nuclear
weapons
claim stance topic
subsidize poor com- munities
poor communitics
motion to topic
claim stance topic
institute a manda-
a mandatory retirement age
motion to topic
tory retirement age
claim stance topic
re-engage with re-engage with myanmar
lower case
Myanmar
claim stance topic
build the Keystone XI. pipeline
the keystone pipeline
motion topic
claim stance topic
Isracl's 2008-2009
israel's 2008-2009 military operations against gaza
lower
military operations
against Gaza
claim stance lopic
build high rises for
high rises for housing
motion to topic
housing
claim stance topic
the blockade of
the blockade of gaza
lower case
Gaza
claim stance topic
Holocaust denial the creation of pri-
holocaust denial
lower case
claim stance topic
the creation of private universities in the uk
lower case
vate universities in the UK
claim stance topic claim stance topic
ASEAN
lower lower case
implement playoffs in collegiate level
implement playoffs in eollegiate level american football
American football
claim stance topic
unleash the free
the free market
motion to topic
market
claim stance topic
have children
children
motion to topic
claim stance topic
huild hydroclectric
hydroelectric dams
motion to topic
dams
Table 7: Label cleaning.

---

dataset argument topic
class name
rephrased class name intelligence tests
rephrased type motion to topic
Intelligence tests bring more
harm than good
agumenn topic
Surrogacy should be banned surrogacy
motion lopic
argument topic
We should ban cosmetic surgery
cosmetic surgery
motion topic
argument lopic
We should abolish capital pun- ishment
capital punistment
motion lopic
argument lopic
should cosmetic surgery for minors
cosmetic surgery minors
molion lopic
argument topic
should han human cloning
human cloning
motion topic
argument topic
We should limit executive com- pensation
executive compensation
motion to topic
agument topic
We should ban naturopathy
naturopathy
motion lopic
argument topic
We should abolish the three-
the three-strikes laws
motion topic
strikes laws
urgument topic
We should legalize organ trade
organ trade
motion topic
argument lopic
should prohibit flag burning should adopt gender-neutral language
flag burning
molion lopic
argument topic
gender-neutral language
motion lopic
argument topic
We should subsidize Wikipedia
wikipedia
motion to topic
argument topic
We should legalize cannabis
cannabis
motion topic
argument topic
We should introduce compul- sory voting
compulsory voling
motion lopic
argument lopic
We should limit judicial
judicial activism
motion lopic
tivism
argument topic
should adopt multi-party
multi-party system
motion topic
system
argument topic
should adopt lihertarianism
lihertarianism homeschooling
motion topic motion to topic
argument topic
Homeschooling should be
banned
Table 8: Label cleaning.
dataset urgument topic
class name
rephrased class name student loans
rephrased type motion to topic
We should subsidize student
loans
argument topic
We should subsidize stay-at-
stay-at-home dads
motion to topic
home dads
argument topic
Payday loans should be banned
payday loans
motion topic
argument topic
Assisted suicide should he
assisted suicide
motion to topic
criminal offence
argument topic
Holocaust denial should be a criminal offence
holocaust denial
motion to topic
argument topic
Social media brings more harm than good
social media
motion to topic
urgument topic
We should ban private military companies
private military companies
motion to topic
argument topic
The use of public defenders
the use of public defenders motion to topic
should be mandatory
argument topic
We should abandon the of
the use of school uniform Foster care brings more harm
motion topic
school uniform
argument topic
care
motion to topic
than good
argument topic
We should ban targeted killing
targeted killing
motion to topic
argument topic
We should fight for the abolition of nuclear weapons
the abolition of nuclear weapons
motion to topic
argument topic
We should ban algorithmic trad- ing
algorithmic trading
motion to topic
argument topic
We should ban whaling
whaling
motion to topic motion to topic
argument topic
The of celibacy should be
the vow of celibacy
abandoned
argument topic
We should legalize prostitution
prostitution
motion to topic motion to topic
argument topic
We should adopt a zero-
zero-tolerance policy in schools
tolerance policy schools
argument topic
We should abolish zoos
2008
motion to topic
argument topic
We should abandon marriage
marriage
motion to topic
Table 9; Label cleaning.

---

dataset
class
rephrased class
rephrased type
argument topic
We should ubandon television
television
motion topic
argument topic
should abolish intellectual
intellectual property rights
to topic
property rights
argument topic
We should end mandatory retire- ment
retirement
motion to topic
argument topic
We should abolish the right to keep and bear arms
the right to keep and bear arms
motion to topic
argument topic
Blockade of the Gaza Strip should be ended
blockade of the gaza strip vocational education
motion to topic
argument topic
We should subsidize vocational education
motion topic
argument topic
We should stop development of autonomous cars
the development of autonomous
topic
argument topic
should the use of child actors
the use child actors
motion to topic
argument topic
We should adopt an austerity regime
austerity regime
motion to topic
argument topic
We should adopt atheism
atheism
motion to topic
argument topic
We should end affirmative ac- tion
athrmativv action
motion to topic
argument topic
We should prohibit women in combat
women in combat
motion topic
argument topic
We should ban the Church
the church scientology
motion topic
Scientology
argument topic
should legulize sex selection
sex selection
motion to topic
argument topic
We should prohibit school
school prayer
motion to topic
prayer
argument topic
Entrapment should he legalized
entrapment legalized
motion to topic motion to topic
argument topic
We should close Guantanamo
guantanamo bay detention camp
Bay detention camp
argument topic
We should ban factory farming
factory farmming
motion topic
argument topic
We should end racial profiling
racial profiling
motion topic
Table 10: Label cleaning.

---

dataset
class name
rephrased class name
rephrased type
argument topic
We should ban telemarketing
telemarketing homeopathy
motion topic
argument topic
Homcopathy brings more harm
motion topic
than good
argument topic
should missionary work
missionary work
motion topic
argument topic
should cancel pride parades
cancel pride parades
motion lopic
argument topic
We should legalize polygamy
polygamy
motion to topic
argument topic
We should abolish sale spaces
sale spaces
motion to lopic
argument topic
We should oppose collectivism
collectivism
motion to topic motion topic
argument topic
We should fight urbanization
fight urbanization
argument topic
We should ban fast food
fast food
motion lopic
argument topic
We should subsidize embryonic rescarch
embryomic cell research
motion topic
argument topic
should subsidize space ex- ploration
space exploration
motion topic
argument topic
We should end the use of eco-
the use of economic sanctions
motion to topic
nomic sanctions
argument topic
We should abolish the Olympic Games
the olympic games
motion to topic
argument topic
We should subsidize journalism translate
journalism translation
motion topic
clinc150
grammatical
150
meaning of_life
meaning life
eleaning
clinc150
insurance change
insurance change
cleaning
clinc150
find find_phome
finding a phone
grammatical
clinc150
travel alert
travel alert
cleaning
clinc150
plo_request
request
cleaning
clinc150
improve_enee citeeeeeeeee_ttteeeeeee score
improving credit score
grammatical
clinc150
Jact
fact
grammatical
150
change language
change_language
changing language
grammatical
cline150
replacement card duration
replacement card duration
cleaning
clinc150
application status
application status
cleaning
Table 11: Label cleaning.

---

dataset
class name
rephrased class name
rephrased type
clinc
light status
flight status
cleaning
clinc 150
flip_coin
fliping coin
grammatical
clinc 150
change user name_mmme
changing user name
grammatical
clinc
where you from
where you from
grammatical
clinc150
shopping list update
shopping list update
cleaning
clinc150
what can ask you
1 can ask you
grammatical
clinc 150
maybe
not being sure
grammatical
cline 150
oil Bil_chang__haa _ow
how to change oil
grammatical
clinc 150
restaurant reservation
restaurant reservation
cleaning
clinc 150
confirm reservation
confirming roservation
grammatical
c5inc150
Ireeze
freezing account
grammatical
c5ine150
rollover_40Ik
rollover 40lk
cleaning
clinc
iito_made_yooe
who made you
cleaning
clinc
user_nane
user name
cleaning
cline 150
next next_song
next song
cleaning
clinc
restaurant suggestion
restaurant suggestion
grammatical
clinc 150
Swardd_balanne
rewards balance
cleaning
clinc150
pay_ bill
paying
grammatical
cline150
spending_histi
spending history
eleaning
cline 150
pto wwrrrrece_ssatteee
pto request status
cleaning
clinc 150
credit score
credit score
cleaning
clinc 150
new new_card
new card
cleaning
clinc 150
Jost Juggage
lost luggage
cleaning
clinc 150
oil_change_ _wee
when change oil
grammatical
clinc150
yes
grammatical
clinc150
travel suggestion
travel suggestion
cleaning
clinc150
todo polo_list_uddate
todo list update
cleaning
clinc
change speed
changing speed
grammatical
clinc 150
tire_pressure
tire pressure
cleaning
clinc 150
negation
grammatical
clinc150
nutrition into
nutrition info
cleaning
clinc
carry_ on
carry
grammatical
clinc150
plo_used plo used
plo used
cleaning
clinc 150
schedule_mainnennnnce maintenance
scheduling maintenance
grammatical
cline 150
travel_ _nt__niiccationn
travel notification
clcaning
clinc 150
sync device
synс device
cleaning
clinc 150
thank
thank
cleaning
clinc
roll dice
roll dice
cleaning
cline
food_las
food expiration
grammatical
clinc 150
cook time
conk time
clcaning
clinc 150
reminder update
reminder update
cleaning
cline 150
report Loport_lostt_oot lost_card card
reporting a lost card
grammatical
Table 12: Label cleaning.

---

dataset
class name
rephrased class
rephrased type
clinc100
ingredient_ substitution
suprechen_ssussiiuutin
ingredient substitution
cleaning
cline
make_call
making call
grammatical
cline
Jist
todo list
cleaning
clinc 150
tlange__ccemn
changing an accent
grammatical
clinc 150
bill due
bill due
cleaning
clinc 150
damaged_com card
damaged card
cleaning
clinc 150
cetaurant_reviewee
restaurant reviews
cleaning
clinc150
jo_youuhaave have__eelsss
question about your pets
grammatical
clinc 150
schedule_mecting
scheduling meeting
grammatical
clinc
gas type
gas type
cleaning
clinc150
plug plug_type
type
plug type
cleaning
clinc 150
tire change
tire change
cleaning
clinc 150
exchange rale
exchange rale
cleaning
clinc 150
_ox__holida
next holiday
cleaning
150
change volume
changing volume
grammatical
clinc150
who_do Siho_do_you_mvee fio__yo__worr_e for
whom you work for
grammatical
clinc
credit limit
credit limit
cleaning
clinc
how_busy
wailing time
grammatical
cline 150
sncept_reservation
accepting reservations
grammatical
clinc 150
order status
order status
cleaning
clinc 150
pin_change
change
cleaning
clinc150
blocked
account blocked
cleaning
clinc150
what_song
what song
cleaning
clinc 150
interatiinal__cn fecs
international fees
cleaning
clinc 150
last maintenance
last maintenance
cleaning
cline 150
meeting
mecting__sceddle schedule
meeting schedule
clcaning
clinc 150
ingredients list
ingredients list
cleaning
clinc 150
report [raud
reporting fraud
grammatical
clinc 150
grcasurement_cooverstte
measurement conversion
eleaning
smart home
smart home
cleaning
clinc
bnok_hotel
booking a hotel
grammatical
clinc 150
current location
current location
cleaning
clinc 150
min_ _aun_p
min payment
cleaning
clinc 150
chisper_mode
whisper mode
cleaning
clinc 150
canceling
cancel
grammatical
clinc150 150
internationa_
international visa
eleaning
pto balance
pto balance
cleaning
clinc150
reset reset_ settings
reset settings
cleaning
cline 150
Dhha_is__our_name
what your name is
grammatical
clinc 150
direct_deposi
direct deposit
cleaning
clinc 150
interest_re rate
interest rate
cleaning
clinc 150
credit limit change
credit limit change
cleaning
clinc150
what_are Shhat_are_your hobbies
what your hobbies are
grammatical
Table 13: Label cleaning.

---

dataset
class
rephrased class
rephrased type
c50
book flight
booking flight
grammatical
clinc150
shopping list
shopping list
cleaning
clinc 150
bill_ ball_balance
bill balance
cleaning
clinc 150
share_lncatinne location
sharing location
grammatical
clinc 150
redeem_ sadeem_rewwads
redeem rewards
cleaning
clinc 150
play music
asking play music
grammatical
150
calendar update
calendar update
cleaning
c5ine150
are_you_a_bol
asking you are
grammatical
c5i
expiration date
expiration
cleaning
clinc 150
update_playlist
updating playlist
grammatical
clinc 150
cancel_ reservation
canceling reservation
grammatical
clinc 150
tell_joke
telling a aoke
grammatical
clinc 150
change_ate [lange_ai_name,
changing name
grammatical
clinc150
how how_old_aree ool__od___eeooo
how old you
grammatical
cline150
car_rental
car
cleaning
clinc 150
jump_start start
jump start
cleaning
clinc150
meal suggestion
meal suggestion
cleaning
clinc 150
order_checks order checks
ordering checks
grammatical
clinc 150
card declinee
declined card
grammatical
cuad
Filename
deletion
cuad
Document Name
delction
cuad
Document Name-Answer
deletion
cuad
Parties
deletion
cuad
Parwies-Asswr
deletion
cuad
Agreement Date
deletion
cuad
Agreement Date-Answer
deletion
cuad
Effective Date
deletion
cuad
Effective Date-Answer
delction
cuad
Expiration Date-Arswer
deletion
cuad
Renewal Term- Semm-Answer
deletion
cuad
Notice Period To Terminate
deletion
Renewal- Answer
cuad
Governing Law-Answer
deletion
cuad
Most Favored
Nation-
deletion
Answer
cuad
Competitive Restriction
deletion
Exception-Answer
cuad
Non- Ca---ooo-eesssse
deletion
cuad
Exclusivity Answer
deletion
cuad
No-Solicit Of Customers-
deletion
Answer
cuad
No-Solicit Of Employees-
deletion
Answer
cuad
Non- Non-Disparagement-
deletion
Answer
cuad
Termination
For
delction
Convenience-Answer
Table 14: Label cleaning.

---

dataset
class
rephrased class
rephrased type
cuad
Rolr/Rolo/Roln Answer
-owweee
deletion
cuad
Change Control-Answer
deletion
cuad
Anti-Asignment-AAne
deletion
cuad
Revenue/Profit Sharing-
deletion
Answer
cuad
Price Restrictions-Are AessreetoossAnsser
deletion
cuad
Minimum Commitment-
deletion
Answer
cuad
Volume Restriction-Answe
delction
cuad
Ip Ownership Assignment-
deletion
Answer
cuad
Joint Ip Ownership-Answer
deletion
cuad
License Grant-Answer
deletion
cuad
Non- Transferable License-
deletion
Answer
cuad
Affiliate License-Licensor-
deletion
Answer
cuad
Affiliate License-Licensen
deletion
Answer
cuad
Unlimited/All- You-Can-
deletion
Fat-L-ccnnse Answer
cuad
Irrevocable Or Perpetual
deletion
License-Answer
cuad
Source Code
Escrow-
deletion
Answer
cuad
Post- Termination Services-
deletion
Answer
cuad
Audit Rights-Ansee Answer
deletion
cuad
Uncapped Rabilltt-Answee
deletion
Cap On Liability-Answer
deletion
Table 15: Label cleaning.

---

dataset
class
rephrased class
rephrased type
cuad
Liquidated
Damages-
deletion
Answer
cuad
Warranty Duration-Answa
deletion
cuad
Insurance-Ausee
deletion
cuad
Covenant Not To
Sue-
deletion
Answer
cuad
Third Party Benefieiary-
deletion
Answer
20 newsgroup
alt atheism
atheism
readability
20 newsgroup
comp graphics
computer graphics
readability
20 newsgroup
comp os ms-win mns-windows misc
microsoft windows
readability
20 newsgroup
comp sys ibm pe hardware
pc hardware
readability
20 newsgroup
comp sys mac hardware
mac hardware
readability
20 newsgroup
comp windows
windows
readability
20 newsgroup
misc forsale
for sale
readability
20 newsgroup
cars
readability
20 newsgroup
motorcycles
motorcycles
readability
20 newsgroup
rec sport bascball
baschall
readability
20 newsgroup
rec sport hockey
hockey
readability
20 newsgroup
sci crypt
cryptography
readability
20 newsgroup
sci electronics
electronics
readability
20 newsgroup
med
medicine
readability
20 newsgroup
space
space
readability
20 newsgroup
soc religion christian
christianity
readability
20 newsgroup
talk politics gums
guns
readability
20 newsgroup
talk politics mideast
middle east
readability
20 newsgroup
talk politics mise
politics
readability
20 newsgroup
talk religion mise
religion
readability
banking77
activate my card
activating my card
grammatical
banking77
age limit
age limit
cleaning
banking77
apple pay or google pay
apple pay or google pay atm support
cleaning
ba
atm support
cleaning
hanking77
automatic top up balance not updated after
automatic top
clcaning
banking77
balance that has not been up- dated after bank transfer
grammatical
bank transfer
banking77
balance not updated after
balance that has not been up- dated after cheque cash de- posit
grammatical
cheque cash deposit
Table 16: Label cleaning.

---

dataset
class name
rephrased class name
rephrased type
banking?7
beneficiary not allowed
heneficiary who is not allowed
grammatical
banking77
cancel transfer
canceling transter
grammatical
banking?7
card about to expire
card that is about to expire
grammatical
hanking77
card acceptance
card acceptance
cleaning
ba
card arrival
card arrival
cleaning
banking77
card delivery estimate
card delivery estimation
grammatical
banking77
card linking
card linking
cleaning
banking?7
card not working
card not working
cleaning
banking77
card payment fee charged
card payment fee that was charged
grammatical
banking77
card payment not recog-
card payment not recognised
cleaning
nised
banking77
card payment wrong card payment wrong exchange
cleaning
change rale
banking77
card swallowed
card swallowed
elcaning
banking77
cash withdrawal charge
cash withdrawal charge
cleaning
banking77
cash withdrawal not recog-
cash withdrawal not recognised
cleaning
nised
banking77
change pin
changing pin
grammatical
banking?7
compromised card
compromised card
cleaning
banking77
contactless working country support
contactless not working
cleaning
banking77
country support
cleaning
ba
declined card payment
declined card payment
eleaning
banking77
declined cash withdrawal
declined cash withdrawal
grammatical
banking7 77
declined transfer direct debit payment not recognised
declined transfer
cleaning
banking77
direct debit payment not recog- nised
cleaning
banking77
disposable card limits
disposable card limits
cleaning
banking77
edit personal details
editing personal details exchange charge exchange rale
grammatical
banking77
exchange charge
eleaning
banking77
exchange rate
cleaning
banking77
exchange via app
exchange via app
cleaning
banking77
extra charge on statement
extra charge on statement
cleaning
banking 77
failed transfer
failed transfer
cleaning
banking
fiat currency support
fiat currency support
cleaning
banking77
gel disposable virtual card
getting disposable virtual card
grammatical
banking77
get physical card
getting physical card
grammatical
banking77
getting spare card
getting spare card
cleaning
banking? 77
getting virtual card
getting virtual card
cleaning
banking7
lost or stolen card
lost or stolen card
cleaning
banking77
lost or stolen phone
lost stolen phone
cleaning
Table 17: Label cleaning.

---

dataset
class name
rephrased class name
rephrased type
banking?7
order physical card
ordering physical card
grammatical
banking77
passende forgotten
forgotten passcode
grammatical
banking 77
pending card payment
pending card payment
cleaning
hanking77
pending cash withdrawal
pending cash withdrawal
cleaning
banking77
pending top up
pending top up
cleaning
banking77
pending transfer
pending transfer
cleaning
banking?7
pin blocked
blocked pin
grammatical
banking?
receiving money
receiving money
cleaning
banking?7
Refund not showing up
refund not showing up
clcaning
banking?7
request refund
refund request
grammatical
ba
reverted card payment?
reverted card payment
grammatical
ba
supported cards and cies
supported cards and currencies
cleaning
banking77
terminate account
terminating account top up by bank transfer charge
grammatical
banking77
top up by bank transfer charge
cleaning
banking77
top by card charge
top up by card charge
cleaning
banking77
top by cash cheque
up by cash cheque
cleaning
banking77
top failed
failed top up
grammatical
banking
top up limits
top up limits
cleaning
banking77
top up reverted
top up reverted
eleaning
banking77
topping up by card transaction charged twice
topping up by card transaction charged twice
cleaning
banking 77
cleaning
banking?7 77
transfer fee charged transfer into account transfer not received by re- cipient
charged transfer fee transferring into account transfer not received by recipi-
grammatical
banking 77
grammatical
banking77
cleaning
banking77
transfer timing
transfer timing
elcaning
banking77
unable to verily identity
being unable verily identity
grammatical
banking77
verify my identity
verifying my identity
grammatical
banking77
verity source of funds
verifying source of funds verilying top up
grammatical
banking77
verily top
grammatical
Table 18: Label cleaning.

---

dataset
class
rephrased class
rephrased type
banking77
virtual card not working
virtual card working visa or mastereard
cleaning
banking77
visa or mastercard
elcaning
banking77
why verify identity
why identity verification is nec- essary
grammatical
banking77
wrong amount of cash re- ceived
wrong amount of cash received
cleaning
banking77
wrong exchange rate for
wrong exchange rate for cash
cleaning
cash withdrawal
withdrawal
dbpedia
Artist
artist
lower case
dbpedia
Plant
plant
lower
dbpedia
Alhum
album
lower case
dbpedia
Animal
animal
lower case
dbpedia
Mean-On Of Transportation
mean of transportation
readability
dbpedia
NaturalPlace
natural place
readability
dbpedia
Athlete
athlete
lower case
dbpedia
OfficeHolder
office holder
readability
dbpedia
Company
company
lower case
dbpedia
Film
film
lower case
dbpedia
Educationnl-Instiiee
educational institution
readability
dbpedia
Written Work
writlen work
readability
dbpedia
Building
building
lower case
dbpedia
Village
village
lower case
lodgar
general
deletion
law stack exchange
contruct-law
contract law
cleaning
stack exchange
constitutional
constitutional law
cleaning
law stack exchange
criminal-
criminal law
cleaning
law stack exchange
tax tax-law
tax law
cleaning
law stack exchange
civil-law
civil law
clcaning
law stack exchange
intellectual- prooperty
intellectual property
cleaning
massive
daletime query
getting date time details
grammatical
massive
lightchange
changing hue light
grammatical
massive
transport licket
getling transport ticket
grammatical
massive
takeaway query
getting takeaway
grammatical
massive
qu stock
stock
grammatical
massive
general greet
greeting
grammatical
massive
recommendation events
event recommendation
grammatical
massive
music dislikeness
music dislikeness
cleaning
massive
turning off
grammatical
Table 19: Label cleaning.

---

dataset
class name
rephrased class name
rephrased type
massive
cooking recipe
cooking recipes
grammatical
massive
currency
currency
grammatical
massive
transport traffic
transport traffic
cleaning
massive
general quirky
quirky issues
grammatical
massive
weather query
the wcather
grammatical
massive
audio volume up
turning up the volume
grammatical
massive
email addcontact
adding email contact
grammatical
massive
takcaway order
takcaway order
grammatical
massive
email querycontact
getting email contact
grammatical
massive
lot huc lightup
incrcasing hue light
grammatical
massive
recommendation locations
location recommendations
grammatical
massive
play audiobook
playing an audio book
grammatical
massive
lists createoradd
creating or adding lists
grammatical
massive
news query
the news
grammatical
massive
alarm query
getting alarm details
grammatical
massive
1ot wemo on
turning wemo on
grammatical
massive
general joke
ajoke
grammatical
massive
qa definition
definitions
grammatical
massive
social query
social media
grammatical
massive
music scttings
music settings
cleaning
massive
audio volume other
audio volume
grammatical
massive
calendar remove
removing from calendar
grammatical
massive
1ot hue lightdim
dimming hue light
grammatical
massive
calendar query
getting calendar details
grammatical
email sendemail
sending en email
grammatical
massive
1ot cleaning
cleaning
grammatical
massive
audio volume down
turning down volume
grammatical
massive
play radio
playing the radio
grammatical
massive
cooking query
cooking details
grammatical
massive
datetime convert
converting date time
grammatical
massive
qa maths
math
grammatical
massive
lightoff
turning off huc light
grammatical
massive
lot lighton
turning on hue light
grammatical
massive
transport query
getting transport details
grammatical
massive
music likeness
music likeness
cleaning
massive
email query
getting email details
grammatical
massive
play music
playing music
grammatical
massive
audio volume mute
muting audio volume
grammatical
massive
social post
posting on social media
grammatical
massive
alarm set
setting an alarm
grammatical
massive
qu factoid
factoids
grammatical
massive
calendar set
setting the calendar
grammatical
Table 20: Label cleaning.

---

datasel
class name
rephrased class name
rephrased type
massive
play game
playing a game
grammatical
massive
alarm remove
removing an alarm
grammatical
massive
lists remove
removing from lists
grammatical
massive
transport taxi
transport taxi
cleaning
massive
recommendation movies
movie recommendations
grammatical
massive
making coffee
grammatical
massive
music query
getting music details
grammatical
massive
play podcasts
playing podcasts
grammatical
massive
lists query
getting lists details
grammatical
ag news
World
world
lower case
ag
Sci/Tech
science and technology
readability
ag
Sports
sports
lower case
ag news
Business
business
lower case
yahoo answers
Sports
sports
lower case
yahoo unswers
Health
health
lower case
yahoo answers
Family & Relationships
family and relationships
readability
yahoo answers
Science & Mathematics
science and mathematics
readability
yahoo answers
Education & Reference
education and reference
readability
yahoo
Entertainment & Music
entertainment and music
readability
yahoo answers
Society Culture
society and culture business and finance,Ao
readability
yahoo answers
Business & Finance
readability
yahoo answers
Politics & Government
politics and goverument computers and internet food and drink
readability
yahoo answers
Computers & Internet
readability
xglue
foodanddrink
readability
sentivent
es r/brand
brand
readability
reuters
pet-chem
deletion
reulers
income
deletion
reuters
strategic-metal
deletion
reuters
lei
deletion
reuters rand
deletion
coconut-oil
deletion
reulers
deletion
reuters oat
deletion
reuters propane
deletion
reuters
saudriyal
deletion
Table 21: Label cleaning.

---

dataset
class
rephrased class
rephrased type
sorghum
deletion
reuters
deletion
reuters
cotton-oil
deletion
reulers
nal-gas
nat gas
cleaning
reuters
fuel
delction
reulers
citruspulp
deletion
reuters
nzdir
deletion
stg
delction
reulers
sun-meal
deletion
reuters
cruzado
deletion
reulers dfl
deletion
reuters
castorseed
deletion
reuters rice
deletion
reuters
comglutenfeed
deletion
cpi
delction
reulers
meal-feed
deletion
reuters
gnp
gross national product
readability
reuters
ship
ships
readability
reuters
acq
acquisition
readability
reuters
barley
deletion
lin-oil
deletion
reuters
com-oil
deletion
reuters
silver
deletion
reuters
soy-meal
deletion
reuters
tapioca
deletion
reulers
orange
deletion
reuters
plywood
deletion
reuters
lead
deletion
deletion
Table 22: Label cleaning.

---

dataset
class name
rephrased class name f'aallle
rephrased type
reulers
I-cattle
cleaning
reuters
linseed
deletion
reuters
pork-helly
delction
reuters
Johs
delction
reuters
naphtha
delction
reulers
rye
deletion
reuters
lumber
deletion
reuters
dkr
deletion
reuters
platinum
deletion
reuters
money-supply
money supply
cleaning
reulers
rape-meal
deletion
reuters
coconut
deletion
reulers
gas
deletion
reuters
heat
deletion
reuters
retuil
deletion
reulers
ipi
deletion
reuters
ringgit
deletion
reuters
copra-cake
deletion
reuters
zine
deletion
reuters
rapeseed
delction
reulers
cpu
deletion
reuters
lishmeal
deletion
reulers
soy-oil
deletion
reuters
veg-oil
vegetable oil
readability
reuters
yen
delction
reulers
carcass
deletion
reuters
lin-meal
delction
reulers
red-bean
deletion
reuters
jet
deletion
reuters
wpi
deletion
reuters
castor-oil
delction
reuters
copper
delction
reulers
wool
deletion
reuters
cocoa
deletion
reuters
groundnut-oil
deletion
reuters
peseta
deletion
reuters
pulm-oil
deletion
reuters
dmk
deletion
reuters
bop
deletion
reuters
I-cattle
delction
reuters
dnst
deletion
reuters
iron-steel
deletion
reulers
reserves
deletion
reuters
rubber
deletion
reuters
rape-oil
deletion
reuters
housing
deletion
reuters
inventories
delction
reulers
potato
deletion
reuters
hog
deletion
reulers
earn
earnings
readability
reuters
skr
deletion
reuters
sun-oil
delction
reulers
palladium
deletion
reuters
sunseed
deletion
reulers
can
deletion
reuters
soybcan
deletion
reuters
cotton
deletion
reulers
austdlr
deletion
reuters
palmkernel
delction
reulers
groundnut
deletion
reuters
alum
delction
reuters
money-fx
money foreign exchange
readability
reuters
nickel
delction
Table 23: Label cleaning.


Reviews:
[{"review_id": "heVDBqIOAg", "reviewer": "Reviewer_PZ6f", "summary": "", "questions": "N/A", "limitations": "", "rating": 2, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for thorough and constructive review.\n\nThank you for highlighting the omitted reference to T0; we will incorporate it into our manuscript.\n\nIn relation to the novelty of our research versus works like T0:\n\nThe aim of most prior work in the field was to create a versatile zero-shot model that can generalize as much as possible, in the sense that it can perform well over a wide range of downstream tasks. We agree that previous work has shown that achieving such a zero-shot model is possible and thus that it is indeed not surprising that this can also be done on narrower scope of tasks like topical text classification. \n\nHowever, our aim here is different: we explore whether honing-in on a **narrower scope** of downstream tasks can enable us to **improve** the zero-shot performance on this type of task only.\n\nAnd since we focus on topical text classification, we propose to focus on fine-tuning the model on this type of tasks only using data from various domains. This enables us to significantly boost the zero-shot performance of the model on topical text classification, compared to the general-purpose models.\n\nTo the best of our knowledge, this approach hasn’t been undertaken before, and it provides a valuable insight for the community, that can be further explored in future work for other tasks as well. \n\nThat said, we concur that this distinction wasn't adequately stressed in the manuscript and will rectify this in our final version.\n\nIn response to the comment on LLMs, we'll refine the terminology in our manuscript. \n\nHowever, we'd like to highlight a few challenges associated with evaluating 65B+ models. \n\nFirstly, assessing the zero-shot performance of the GPT family is challenging due to its undisclosed training data, potentially encompassing supervised training from our evaluation datasets. Secondly, given its paid nature, extensive inference at the paper's scale is not only costly but also non-reproducible should the service evolve in the future. Regarding the Llama family, our trials with Llama in zero-shot on TTC-23 yielded unsatisfactory results, leading to its exclusion. These points will be incorporated into the revised manuscript.\n\nAs for the prompts, initial tests on the dev set affirmed their superiority over other prompts, including those for topical text classification from the flan collection. We'll detail these results upon the paper's acceptance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The methodology is convincing, and the improvement (4% average over 23 datasets) is noticeable. New dataset is assembled.", "reasons_to_reject": "Still the results are not surprising considering that this phenomenon was already observed in a more general (not limited to topical classification) context in the following paper: Multitask Prompted Training Enables Zero-Shot Task Generalization. Victor Sanh, et al., ICLR 2022, which should have been referenced here.\n\nAlso, in my observation, 11B model is not yet called large LM, which is normally reserved for 65B+ (starting with Llama and including GPT3+, Palm and a few others of that size).\n\nSince the prompts are important in this context, an ablation study and sensitivity to the prompts could be have been explored in more depths.\n", "questions_to_authors": "N/A", "paper_topic_and_main_contributions": "The paper demonstrates that fine-tuning on multiple topical text classification datasets improves zero-shot classification, as tested with 11B LM.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Multitask Prompted Training Enables Zero-Shot Task Generalization. Victor Sanh, et al., ICLR 2022.", "Typos_Grammar_Style_and_Presentation_Improvements": "No specific issues. But the content of the paper can be better suited for a journal or a short paper considering limited novelty of the main result."}, "extra_text": {}}, {"review_id": "kYSWkKso2q", "reviewer": "Reviewer_UZow", "summary": "", "questions": "A. How fair is the comparison of the models? Is there a way to conclude about the effect of data used during pre-training of the models at hand on their performance?\n\nB. If you could provide more details about the decoding strategy of the models, it would be very helpful to see if there is an important effect of this configuration on their performance in TTC.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's feedback.\n\nRegarding the potential reproducibility, as stated in the manuscript, upon acceptance, we commit to share the code, data, and models in their entirety, to ensure our flow is clearly reproduceable. We believe that this follows standard practice in the field.\n\nFor the decoding strategy, we employed greedy generation. Given the objective of classification tasks is to identify the class with maximum model confidence, a greedy approach is apt. Factors like creativity and diversity, pivotal for generative tasks, are secondary here. We'll address this in the manuscript.\n\nTypically, hyper-parameters cited in our paper are borrowed from analogous hugging-face experiments. Though deeper examination of their influence is possible, our study currently focuses on other aspects.\n\nWe, too, recognize the importance of data's role in pre-training / training versus testing. Our paper has made strides in this direction via: 1) Assessing the effect of class similarities in training and test data for fine-tuned folds (section 6.4); 2) Omitting datasets trained on by flan and the QA models; 3) Evaluating folds that distinctly differ between domains (section 6.3). A thorough analysis of data impact during LLMs' pre-training is intricate and was reserved due to space and time constraints.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "-The paper exhibits a high level of organization and clarity, effectively conveying the proposed methodology.\n- Comprehensive experiments involving various pre-trained models are conducted across diverse datasets, contributing to the paper's robustness.\n-The integration of tables, summaries, and graphs enhances the reader's comprehension of the paper's content.\n-The model's focus on addressing Topical Text Classification (TTC) aligns with a pertinent challenge. The authors' intention to sharing the benchmark and models with the public is noted.\n-This approach holds potential utility not only within the realm of NLP but also extends to other domains where zero-shot classification finds applicability.", "reasons_to_reject": "- The data, code, fine-tuned models and methodology have not been made public in the current version of the paper. While the authors have expressed their intention to release these resources, this currently hinders the ability to verify the stated claims, results, and procedures, or to replicate the conducted experiments.\n- Further elaboration on the hyperparameters would have enhanced the comprehension of the study's nuances. For instance, details about the decoding strategy, such as whether beam search or sampling techniques were employed, are lacking. This aspect could influence both the model's performance and the runtime during inference.\n- The potential impact of the pre-training data on the models' performance in the TTC task received limited discussion. Considering whether certain models were previously exposed to similar texts or tasks during their pre-training could partially explain performance gaps between the models.\n- An exploration of the models' strengths and weaknesses at the dataset level would have been valuable. The addition of metrics like standard deviation, minimum, and maximum values might have revealed patterns that could guide the scope of future research approaches", "questions_to_authors": "A. How fair is the comparison of the models? Is there a way to conclude about the effect of data used during pre-training of the models at hand on their performance?\n\nB. If you could provide more details about the decoding strategy of the models, it would be very helpful to see if there is an important effect of this configuration on their performance in TTC.", "paper_topic_and_main_contributions": "The paper focuses on Topical Text Classification (TTC), an established research area within natural language processing. The authors address the question of how well large language models (LLMs) can perform in a zero-shot scenario for TTC, considering the recent advancements in these models. They introduce a comprehensive study that evaluates the zero-shot performance of various LLMs on a benchmark called TTC23 , which consists of 23 publicly available TTC datasets spanning diverse domains and styles.\n\nThrough their benchmark and experiments, the authors illustrate the potential of specialized LLMs and provide insights into the practical use of these models for TTC tasks. The work contributes to advancing the understanding of the capabilities and limitations of LLMs in zero-shot TTC scenarios, providing guidance for practitioners interested in this field.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "-", "Typos_Grammar_Style_and_Presentation_Improvements": "-"}, "extra_text": {}}, {"review_id": "5wz5r2Uuv3", "reviewer": "Reviewer_aRE1", "summary": "", "questions": "- Given the trend of not open-sourcing the dataset used to train LLMs, would the proposed dataset and evaluation method applicable to good amount of LLMs?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insights.\n\nOn the topic of LLMs, several challenges arise when evaluating them in our context. \n\nFirstly, the training methodology of chatGPT/GPT4 remains undisclosed. Thus, we can’t know if any of the datasets evaluated in our paper were used for supervised training by the GPT models. This is different from flan-t5 / t5  where the datasets used for flan's training are identified (https://github.com/google-research/FLAN/blob/main/flan/v2/flan_collection_info.csv), and t5's pre-training data is public and unsupervised (https://huggingface.co/datasets/c4). \n\nSecondly, accessing these models through a service poses reproducibility issues if the service evolves in the future. The associated financial implications of using this paid service, especially given our paper's scale, further complicate matters. As for the Llama models, our initial attempts with Llama in zero-shot scenarios underperformed, leading to its omission. These nuances will be elaborated upon in the revised manuscript. \n\nFurthermore, our results indicates that the benefits of fine-tuning on topical datasets span various model sizes, ranging from 355M to 11B. This implies that the trend may continue for larger models.\n\nIn-context learning is a viable route. Yet, from a user's viewpoint, once fine-tuning is executed on a separate dataset from the test data, the distinction between a fine-tuned model and an original becomes imperceptible, leading to consistent inference efforts and costs. Conversely, in-context learning demands user intervention in curating few-shot examples. Hence, we chose to spotlight the zero-shot paradigm, given its potential appeal in real-world applications\n\nRegarding the trend mentioned by the reviewer, of not open-sourcing the datasets used to train LLMs  - while we concur that such a tendency is emerging, we remain optimistic about the sustained demand for open-source datasets and the associated LLMs, that will maintain the relevance and importance of the results and paradigms presented in our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper presents the first comprehensive benchmark (TTC23) to evaluate zero-shot topical text classification, covering a diverse range of datasets. The proposed dataset and evaluation approach will facilitate future research.\n\n- This paper provides an extensive empirical comparison of various methods for zero-shot TTC. The paper demonstrates that fine-tuning large LMs like Flan-T5 on TTC data significantly improves their zero-shot performance on unseen topics.\n\n- This paper also presents experiments on the effect of label semantics, and inference speed tradeoffs, providing useful insights.", "reasons_to_reject": "- This paper does not evaluate powerful recent LLMs like ChatGPT, over the Llama Families. While the author argues the dataset used to train those models are unclear, it is not convincing not trying them (Especially given the training data for T5 base model is also unclear). Given the missed SOTA LLMs could be competitive or superior for zero-shot TTC, this Limits the generality of conclusions.\n\n- The largest model tries in this paper contains 13B parameters, which is still rather a small one without much \"emergence\" ability. With powerful LLMs, zero-shot or ICL using selective examples from the training subset could be an even cheaper approach as comparing to fine-tuning, which is missing in the paper's experiments or discussion.\n\n", "questions_to_authors": "- Given the trend of not open-sourcing the dataset used to train LLMs, would the proposed dataset and evaluation method applicable to good amount of LLMs?", "paper_topic_and_main_contributions": "The paper presents a systematic study of the performance of large language models on zero-shot topical text classification (TTC) over a diverse benchmark of 23 datasets (TTC23). In this paper, the author compares several approaches to zero-shot TTC including similarity-based models, QA-based models, NLI-based models, and instruction-tuned models. It also proposes training TTC models by fine-tuning on existing labeled TTC datasets. The proposed TTC23 dataset includes 23 existing datasets covering diverse domains and styles. The paper preprocesses them into a unified format and cleanses labels for consistency. The paper conducts experiments comparing aforementioned methods on TTC23 and found out fine-tuning on TTC23 significantly improves zero-shot performance over using off-the-shelf models. Among the LLMs tried, Flan-T5-XXL FT obtains the top results. The author concludes LLMs exhibit strong zero-shot TTC capabilities, and fine-tuning on diverse TTC data can further enhance performance on new unseen categories.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a benchmark for topical text classification called TTC23, which is curated from a diverse set of existing datasets for the task. The authors use this benchmark to explore the zero-shot capabilities of LLMs for the task, and also show that fine-tuning on diverse TTC data can further enhance performance on new unseen categories.

Reviewers agree that the proposed benchmark is comprehensive, and that the set of experiments presented is extensive, involving various pre-trained models. That will benefit "understanding the capabilities and limitations of LLMs in zero-shot TTC scenarios, providing guidance for practitioners interested in this field."

One of the concerns raised by the reviewers involves the novelty of the results, since similar conclusions have been obtained previously in more general settings. The authors argue that their contribution lays in providing in-depth insights in a narrower set of downstream tasks, which improves zero-shot performance in TTC in unseen ones. Another concern is regarding not comparing against ICL models, which the authors argue that, while relevant, does not align with their focus on allowing end users to only rely on zero-shot interactions with good performance. Authors are encouraged to include their valid justifications for not comparing against GPT models, as well as their clarifications for the effects of data used during pre-training and testing.
Decision: Accept-Findings