Semantic matching for text classification with complex class descriptions
Authors: Brian M De Silva
Kuan-Wen Huang
Gwang Gook Lee
Karen Hovsepian
Yan Xu
Mingwei Shen
Venue: EMNLP 2023 Main (2023)
Keywords: natural language processing
few-shot learning
zero-shot learning
semantic matching
TL;DR: We show that a matching approach (matching class descriptions with examples) is a cost-effective way to handle new classes in text classification problems with complex class descriptions.

Abstract: Text classifiers are an indispensable tool for machine learning practitioners, but adapting them to new classes is expensive. To reduce the cost of new classes, previous work exploits class descriptions and/or labels from existing classes. However, these approaches leave a gap in the model development cycle as they support either zero- or few-shot learning, but not both. Existing classifiers either do not work on zero-shot problems, or fail to improve much with few-shot labels. Further, prior work is aimed at concise class descriptions, which may be insufficient for complex classes. We overcome these shortcomings by casting text classification as a matching problem, where a model matches examples with relevant class descriptions. This formulation lets us leverage labels and complex class descriptions to perform zero- and few-shot learning on new classes. We compare this approach with numerous baselines on text classification tasks with complex class descriptions and find that it achieves strong zero-shot performance and scales well with few-shot samples, beating strong baselines by 22.48% (average precision) in the 10-shot setting. Furthermore, we extend the popular Model-Agnostic Meta-Learning algorithm to the zero-shot matching setting and show it improves zero-shot performance by 4.29%. Our results show that expressing text classification as a matching problem is a cost-effective way to address new classes. This strategy enables zero-shot learning for cold-start scenarios and few-shot learning so the model can improve until it is capable enough to deploy.

Semantic matching for text classification with complex class descriptions
Brian M. de Silva
Kuan-Wen Huang Gwang Gook Lee
Karen Hovsepian Yan Xu Mingwei Shen
Amazon (slvbria, kuanwen, gglee, khhovsep, yanxuml, mingweis)@amazon.co com
Abstract
into the model incurs substantial costs. The process often involves labeling new examples, relabeling existing examples, and retraining the model. To reduce these costs, we seek a method which rapidly adapts from seen classes to new/unseen ones, using few to no labeled examples.
Text classifiers are indispensable tool for machine learning practitioners, but adapting them new classes expensive. To reduce the cost of new classes, previous work exploits class descriptions and/or labels from existing classes. However, these approaches leave gap in the model development cycle as they support either zero- or few-shot learning but not both. Existing classifiers either do not work shot problems, or fail to improve much with few-shot labels. Further, prior work aimed at concise class descriptions, which may be insuf- ficient for complex classes. We overcome these shortcomings by casting classification as a matching problem, where a model matches examples with relevant class descriptions. This formulation lets us leverage labels and complex class descriptions to perform zero- and few- shot learning on new classes. We compare this approach with numerous baselines text clas- sification tasks with complex class descriptions and find that achieves strong zero-sho perfor- mance and scales well with few-shot samples, beating strong baselines by 22.48% (average precision) in the 10-shot setting. Furthermore, we extend the popular Learning algorithm to the zero-shot matching setting and show improves zero-shot perfor- mance by 4.29%. Our results show that ex- pressing text classification as a matching prob- lem is cost-effective way to address new classes. This strategy enables zero-shot learn- ing for cold-start scenarios and few-shot learn- ing so the model can improve until it is capable enough to deploy.
We study a common scenario wherein we have labeled data for a multi-class multi-label text classification problem and then new classes are introduced. Crucially, assume access to class descriptions. Such information not uncommon in applications with frequent class changes. For example, in product compliance, new classes are often accompanied by documentation describing the products covered by the new regulations. Con- cise descriptions may suffice for simple classes like books, but detailed descriptions are needed for more specific complicated classes like books from genre I or genre or ... or genre 10. Given labeled data and descriptions for existing classes, and only descriptions for new classes, our goal to build a model to classify examples into new classes with minimal additional labeling, making it deploy- able in read-world scenarios. We focus on mod- est sized language models rather than large ones since they more readily scale the large problems common in real-world applications. This paper addresses three problems in this setting: how (1) leverage both class descriptions and existing la- beled data to generalize to new classes (in zero- and few-shot settings); (2) effectively use long, com- plex class descriptions; and (3) align training with zero-shot evaluation.
Meta-
1 Introduction
First, to generalize from seen classes to unseen ones, an ideal approach should exploit all available data sources and be applicable in both zero- and few-shot settings. Intuitively, model which incor- porates more information should be more accurate. Labels from existing classes help language models adapt from pre- pre-training objectives to the problem domain, while class descriptions aid in generaliz-
Advances in supervised text classification have en- abled the automation of countless classification pipelines, but such methods struggle when new classes are introduced. In the typical setting, labels are collected for a fixed number of classes, and a model is trained to assign examples to their respec- tive classes. However, incorporating new classes

---

matching score (z) E [0,
formation in long, complex class descriptions, we use cross-cncoder architecture with a pre-trained transformer backbone. This approach facilitates rich interaction between example text and class de- scriptions by ingesting them together as a single concatenated input.
Matching Model
CCS
example text
|SEPI
scction
|SEP
section
|SEP]
class description ey
Figure 1: A matching model outputs matching score f(z;, oj) by assessing whether example text Z "matches" class description Cj.
Third, the matching formulation described above unifies zero- and few-shot learning, but it still ex- hibits a discrepancy between training and evalua- tion. Matching model training follows the standard supervised learning paradigm: training loss is com- puted on seen classes and descriptions in the hope that the model will generalize to unseen descrip- tions. We propose a novel extension of Model- agnostic meta-learning (MAML) (Finn et al., 2017) to the matching setting, specifically targeting zero- shot learning. Similarly to how MAML signed align model training to few-shot evalua- tion, our extension augments the training process to better mimic zero-shot evaluation.
ing to new classes with fewer labels. Flexibility between zero- and few-shot learning is important for ensuring the model works when no or few labels are available. Additionally, zero-shot performance is often insufficient to meet real-world performance requirements, making it desirable for the model to significantly improve with more collected labels. Existing techniques for zero- and few-shot learn- ing tend satisfy only some of these desiderata. They either specialize in only zero- or few-shot settings, or they exploit only one of the aforemen- tioned data sources. To address this, we follow Yin et al. 2019; Halder et al. 2020; Wang et al. 2021 and reformulate multi-class or multi-label text clas- sification as binary matching problem, where we train model to match example texts with class de- scriptions based semantic similarity. The model assigns each (example text, class description) pair a score, representing the likelihood of the example belonging to the respective class as in Figure 1. We use seen class labels to teach the model to match ex- amples with applicable class descriptions. For the zero-shot setting we simply pass in descriptions for new classes, and for few-shot setting we continue to fine-tune the model on labels and descriptions from new classes. Therefore this approach uses (a) class descriptions and existing labels to perform zero-shot classification and (b) additional labels from new classes to accomplish few-shot learning.
de-
In this paper we investigate the efficacy of the matching approach using cross-encoder architec- ture for zero- and few-shot text classification with long, complex class descriptions. Our contribu- tions are follows:
We show that the matching model effectively leverages existing labels and complex class descriptions to reduce the label cost of adapt- ing to new classes: it exhibits strong zero- shot performance (13.30% better than 10- shot BERT) and outperforms the baselines by 22.48% (macro average precision) the 10-shot setting.
We extend MAML from few- to zero-shot learning to improve zero-shot performance for matching models by 4.3%.
The remainder of this paper is structured as fol- lows. We review related work in Section 2. Sec- tion 3 formalizes our problem statement, introduces the matching model and baselines, and details our extension of MAML to zero-shot learning. In Sec- tion 4 we discuss our experiment setup then present and interpret the results of our experiments. We give closing thoughts in Section 5.
Second, we seek model which is effective for long, complex class descriptions, but this area is relatively unexplored. Previous work (Yin et al., 2019; Wang et al., 2021; Schick and Schütze, 2021; Pappas and Henderson, 2019; Pushp and Srivas- tava, 2017; Xiao et al., 2019; Zhang et al., 2019) has primarily focused on short inputs such as class names brief sentences. They work when classes can be concisely summarized in few words (e.g. bicycle or socks) but may be inadequate when class descriptions are necessarily long or specific (c.g. Category I equipment). To fully leverage the in-
Related work
Few-shot learning Few-shot learning was orig- inally explored in the domain of image classifica- tion (Koch et al., 2015; Vinyals et al., 2016; Snell

---

al., 2017: Bateni al., 2022) where a subset of classes- train have sufficient examples, while the remaining classes-test classes- have relatively few examples. Such methods have since been repurposed for NLP tasks (Dopierre et al., 2021), and new techniques, such as in-context and prompt-based learning, have shown great promise for few-shot text classification. Large language models like GPT-3 and 4 (Brown et al., 2020) can solve NLP tasks by including few-shot examples in- context. However, their large parameter sizes and maximum token limitations hinder their ability to efficiently utilize more labels. Prompt-based meth- ods like pattern exploiting training (PET) (Schick and Schütze, 2021, 2020) address these challenges by prompt-tuning language models and using them to annotate large unlabeled datasets for downstream tasks. However, PET's dependence on domain- specific data and handcrafted patterns limits its ap- plicability. Methods like LM-BFF seek to automate prompt selection, but they are limited to cases when classes can be described in a few tokens (Gao et al., 2021a). SetFit (Tunstall et al., 2022) addresses the shortcomings of prompt-based methods by fine- tuning a sentence transformer in contrastive manner with a prompt-free framework. Finally, adapters (Houlsby et al., 2019; Pfeiffer et al., b020a.b) have been shown to perform well in sew-shot settings be- cause of their ability to preserve the generalizability of pre-trained transformers while avoiding catas- trophic forgetting and overfitting, In this work we test methods which do not require tailored prompts: prototypical networks, adapters, and SetFit.
be categorized into a taxonomy (Trabelsi et al., 2021; Khattab and Zaharia, 2020) based on the amount of interaction they allow between input pairs. In increasing order of interaction, perfor- mance, and computational complexity, we have (1) representation-basse models (or bi-encoders) (Pap- pas and Henderson, 2019; Pushp and Srivastava, 2017). which embed the inputs separately before computing their matching score; (2) late interaction models (Trabelsi al., 2021; Khattab and Zaharia, 2020), which allow more interplay between input pairs, but still have separate encoding stacks; and (3) all-to-all models (or cross-encoders, (Ye et al., 2020; Yin et al., 2019; Halder et al., 2020), which facilitate immediate interaction. Matching models also have close ties to information retrieval (see Appendix A).
Label-aware classification Label-aware classi- fication techniques enhance classifiers by incorpo- rating additional label information alongside stan- dard input features. These methods use various types of label information, including label hierar- chy (Nam, 2019), label dependency or correlation (Zhang et al., 2018; Yang et al., 2018; Xun et al., 2020), or label semantic information. The datasets in this paper lack label hierarchies, and label corre- lation is not practical in zero- and few-shot settings due to the need for data to learn the correlations. However, label semantic information is useful for zero- and few-shot applications. In the absence of labeled examples, models require some sort of in- formation to scale to new classes. Early approaches gravitated toward representation-be based or late inter- action models and usually assumed short label de- scriptions (Xiao et al., 2019). Recent work (Schick and Schütze, 2021; Pappas and Henderson, 2019; Pushp and Srivastava, 2017; Nayak and Bach, 2020; Ye al., 2021) has shown the promise of using textual entailment as a format for data-pot data-poor NLP tasks. These methods convert NLP tasks into en- tailment problems then leverage trained entailment models. This approach has yet to be explored for datasets with long class descriptions. Other label- aware classification methods use combinations of different types of label information, such as com- monsense knowledge graphs for labels (Nayak and Bach, 2020); heterogeneous label graphs (Chen et al., 2017: Xu and Lapata, 2020); and word em- beddings, class descriptions, class hicrarchy, and knowledge graphs (Zhang et al., 2019).
Zero-shot learning Zero-shot learning for text classification involves classifying texts into classes which were not seen during the learning stage (un- seen classes). To this end, zero-shot techniques often rely transferring knowledge from seen to unseen classes by leveraging semantic attributes or descriptions of classes (Meng et al., 2020), inter- class correlations (Rios and Kavuluru, 2018), or joint embeddings of classes and examples (Nam et al., 2016; Schopf et al., 2022). Given the com- plexity of class descriptions in our datasets, we fo- cus on methods that use semantic attributes. Match- ing models, one such approach, are of particular interest. These models take pairs of texts as input- in our case a class description and example text- and output score. They differ from "matching networks" (Vinyals et al., 2016), which are instead aimed at sho learning. Matching models can

---

3 Models
means that a classifier is only trained on the seen label set Ys without seeing any examples from the unseen label set Yu. That is to say, the train- ing set Duain is a subset of X x Ys, whereas the test set Drest is a subset of X x Y, where Y = Ys + Yu = {0, and YU = {0, 13 with q O. We refer to the new classes un- seen classes since they are not seen during training. The others are seen classes. Note that in order build triplet version of this dataset we require class descriptions (ppmm)m= for classes that are introduced at evaluation time.
Matching differs from standard supervised learning because it uses class descriptions model inputs, in addition to other features. We formalize the matching problem in Section 3.1. Next we describe our proposed solution in Section 3.2: a matching model that measures semantic similarity between class descriptions and examples. Finally, we dis- cuss the baselines against which we compare the matching model in Section 3.3.
3.1 Problem formulation
In Perez et al. 2021, the authors show that prior work tends overestimate few-shot performance due hyperparameters and prompt selections tuned on held-out samples. Therefore, we follow the "true" few-shot learning setting. This means that validations sets contain no unseen class labels. Few-shot learning involves first training a classifier on Dimin C X x Ys. then updating the classifier using few-shot dataset Dew C X x Y before evaluating it test dataset Diest C X Y with the new classes. Here, k-shot learning indicates that D tew-shot contains i positive examples of each new class.
Multi-label and multi-class classification problems can be converted to matching problems and vice- versa. For a multi-label classification task with p labels, we learn function h X - Ys {0, 1 from a multi-label training set Dirin {(1,, 1a where 2; represents an input and My is a p-dimensional binary vector. To general- ize to a matching problem, our objective shifts to learning a function f X x C + [0, 1), where X denotes a text-based instance space and C' a set of class descriptions. Here ff maps (example, class description) pairs to probability scores. This change requires expanding the dataset Dirain into a triplet format: Dusin {( Cj Mty),, where (xs; Cj. 9i.) E XxCx (0, for 1,2,....
3.2 Matching model
and j = 1, p.. In this formulation, instead of predicting the entire multi-label yı. models predict a single binary label Vij for class j. The proba- bility scores fxx 'y) can be converted back into multi-label predictions via thresholding (Zhang and Zhou, 2013), i.e., 0: = [1[f 0y) > %)ll = ., p) for set of thresholds { where 1 is the indicator function.
A matching model (Figure 1) learns semantic re- lationships between examples and class descrip- tions. We interpret its output f (y) as a match- ing score between example and class descrip- tion Cj. During training, we teach the model to relate class descriptions with examples that, dur- ing testing, the model can classify examples into unseen classes by matching examples with corre- sponding (potentially new) class deseriptions.
By viewing a multi-class problem as a con- strained multi-label problem, we can convert it to or from a matching problem. To do so we replace the multi-class target J < {1, 2, ........... .P) with a multi-label target Vi = [1g
Our model design largely follows previous works (Schick and Schütze, 2021; Pappas and Henderson, 2019; Pushp and Srivastava, 2017). For comparability with prior research, we adopt a pre-trained BERT model (Devlin et al., 2018) as our base matching model, though other architec- tures can augment performance (see Appendix E.2, where we show larger models increase perfor- mance). We express the input to the model (#4, Gj) as "[CLS] x, [SEP] çj [SEP]". When class descrip- tions have multiple sections we use [SEP] tokens to separate them, e.g. "[CLS] 24 [SEP] ej section [SEP] 6j section 2 [SEP]." Including the [SEP] token allows the model to distinguish between sec- tions. We use the hidden vector for [CLS] in the
1], MG 2]. ... 11 [g. p))- Then, to get class predic- tion, we take the argmax of f over all classes, i.c.,
i = f Gj).
One drawback of the matching formulation is that the computational complexity of inference scales linearly with the number of examples N and the number of classes p: O(Np). Standard classi- fiers' computational complexities depend only on the number of examples: ((N).
We evaluate the proposed solution and base- lines in shoo and sho-shot settings. Zero-shot

---

final layer as the aggregate representation of the input: g( oy). We then apply a dropout layer, a linear layer, and a softmax to obtain matching score fa;, oj). In the ranking model taxonomy of Siblini et al. 2020; Nam 2019 this is considered an all-to-all interaction or cross-encoder model since BERT's attention mechanism facilitates immediate interaction between example and class description texts.
eral models are fine-tuned for a fixed number of iterations on one support set out of the batch of tasks selected from this episode: few-shot training samples from a subset of the classes The fine- tuned model is then tested on the query ser-test samples from the classes in the support set-and the parameter initialization is updated for the next episode: 0, + Os+1. Together, a support and query set make up task. Typically episodes consist of batches of tasks, which are resampled at the start of each episode. Bringing training and testing into agreement results in a parameter initialization that is primed for few-shot learning. It is natural to wonder whether we can improve matching model performance further with MAML.
For zero-shot classification of an example z; we use the model trained on Dirain to compute match- ing scores between Ty and unseen class descrip- tions f( Cp+1), [Ti (pt2)...., ('ss (p+g) in addition to matching scores on the seen class de- scriptions fZ2 c1); (2),
cp)- To adapt model trained on Dtrain new classes with few-shot data, we fine-tune the model Diew in Wang et al. 2021. The matching model is applicable to both multi-class and multi-label prob- lems. And it can generalize to any number of new classes in the zero-shot setting via additional class descriptions. However, there are likely practical constraints on the number of classes it can learn during fine-tuning. We leave the question of how many future work.
The standard MAML classification framework limits meta-learning to the encoder layers sitting below the classification head, which is task-specific. The classification head is replaced and randomly initialized for each new task in MAML's inner loop, and then frozen in the outer loop. By formulat- ing text classification as a binary "class-to-text" matching task, we can use the same classification head across new tasks, and, therefore, expand meta- learning through the classification head, allowing the MAML framework to improve zero-shot perfor- mance. In particular, recall that the MAML training algorithm proceeds by sampling, at each episode, a set of n support classes S, a set of query classes Q, and labeled samples for each class in and Q. The few-shot application and the need to randomly initialize the classification head for each new task enforce S = Q so that the n-class classification head learned during the inner loop (on classes in S) can be used during the outer loop (on classes in Q). In our matching MAML formulation, this restric- tion is relaxed the query set can consist of new unseen classes, which corresponds to directly evalu- ating the performance under the zero-shot scenario. We note that this generalization is not dependent on the model architecture. It is dependent on our ability to formulate the text classification task as a binary matching task.
This approach is closely related the one pro- posed in Yin et al. 2019 (and, later, Wang et al. 2021; Halder et al. 2020), except for two key dif- ferences. We extend the maximum token length of 128 in Yin et al. 2019 to 512 to account for our datasets' longer class descriptions. We also skip pre-training the MNLI dataset (Williams et al., 2018) since our initial experiments showed it harms performance on our datasets.
3.2.1 Meta-learning
Model-agnostic meta-learning (MAML) (Finn et al., 2017) is popular algorithm for meta- learning neural networks. It is highly versatile, and has been applied classification in vision and NLP, and regression and reinforcement learn- ing problems (Lee et al., 2022; van der Heijden et al., 2021). MAML attempts to learn ideal pa- rameter initializations of arbitrary neural networks for rapid improvement on new tasks with gradi- ent descent optimization, making it ideal for few- shot learning. This is accomplished by aligning the training objective with few-shot performance on randomly chosen downstream tasks. Training proceeds in series of episodes; each episode simu- lating several few-shot learning problems. Starting from the current parameter initialization 6,, sev-
We propose two novel ways of sampling the query set classes in MAML: (1) Local zero-shor: the query and support set classes do not overlap within a given task but may overlap across tasks; (2) Global zero-shot: query set classes and support set classes have no overlap. Both are aimed at improving zero-shot performance by mimicking zero-shot evaluation during training. Each is a

---

different way of enforcing S Q 9. as opposed the usual constraint S Q.
addition to the baselines above, we also compare Lb12Transformervee (Schopf et al., 2022) against a matching model in zero-shot setting (see Ap- pendix E.3).
3.3 Baselines
We consider four baselines: two standard super- vised models and two few-shot approaches. Un- like the matching model, the baselines do not con- sider the class descriptions. Therefore, they tackle classification in the usual way rather than refor- mulating it as matching problem. The super- vised models consist of gradient boosted trees via LightGBM (Ke et al., 2017) and BERT classi- fier (Devlin et al., 2018). Both popular tools for text classification. We consider two few-shot classifiers: BERT model with adapter modules (BERT + Adapter) (Houlsby et al., 2019) and Set- Fit (Tunstall et al., 2022). BERT + Adapter inserts lightweight adapter layers with bottleneck archi- tectures in each transformer block of pre-trained downstream
Experiments
We conduct two experiments test the models from Section 3 zero- and few-shot scenarios. Our primary experiment compares the ability of each model to adapt to new classes. We then explore whether we can improve the zero- or few-shot per- formance of the matching model by incorporating meta-learning during training. Both experiments involve four text classification datasets: two public and two internal real-world e-commerce datasets. The experiment setup is similar in each case. We first select 20% of classes to be unseen, simulating new classes. Next we partition the data into train- ing, few-shot, and test sets consisting of examples from seen, unseen, and seen and unseen classes, respectively. We train the models to predict seen classes on the train sets, update them to predict un- classes using the few-shot sets, and evaluate them on the test sets. In order to measure the speed at which the models adapt unseen classes we vary the number of labels available the few-shot sets as 10, 50, 100, 500, and 1000. The few-shot samples are nested, e.g. 100-shot dataset is a subset of the 500- and 1000- shot sets.
BERT model. When fine-tuning task, only train the weights of adapter layers. The small number of trainable parameters acts as a regularizer to prevent overfitting. SetFit (Tun- stall et al., 2022) is a recently proposed method that uses a two-stage training approach to achieve strong few-shot performance without prompts or verbalizers. It first the-tunes> pre- pre-trained sentence transformer on contrastive pairs sampled from the few-shot set, then it trains classification head on the resulting embeddings. See Appendix C for more information about each model.
Section 4.1 describes the four datasets. Sec- tions 4.2 and 3.2.1 present the results of our model comparison and meta-learning experiments, re- spectively, and discuss their implications. Ap- pendix gives details on how we constructed
Beyond the above baselines, we also gener- alized prototypical networks (Snell et al., 2017) the multi-label setting (Multi-Label Prototypi- cal Classifer- MLPC uses labeled ex- amples from the seen classes to learn metric space in which to classify new examples by com- parison against class prototypes. Each class has two one for positive (in-class) exam- ples and one for negative (out-of-class) examples- formed by averaging the embeddings of fhw-shot positive negative instances of each class. As MLPC is novel, we give more details in Ap- pendix B.
D Dirain- Diew-shols Dust and their triplct counterparts.
4.1 Setup
We tested the matching model and baselines from Section 3 on four text classification datasets: Yahoo Answers (Yahoo) (Zhang et al., 2015), Amazon Re- View Data 2018 (Amazon) (Ni et al., 2019), Product Classification 1 (PC1), and Product Classification 2 (PC2). Yahoo and Amazon are public datasets and PCI and PC2 are internal. Amazon, PC1, and PC2 are all e-commerce product classification datasets. We describe each dataset in more detail below and put additional information in Appendix D.
Besides prototypical networks, which necessi- tate labeled examples for class representations, there are also unsupervised similarity-based meth- ods such Haj-Yahia al. 2019; Schopf al. 2022. They perform classification based on simi- larity between class and example representations, relying on keywords to describe each class. In
Few-shot Yahoo (Yahoo) The Yahoo Answers dataset involves a multi-class text classification task: to classify historical Q&A data from the Ya- hoo Answers website into one of 10 categories

---

based on the question and top answer. The labels are noisy as they were assigned by the original ques- tion posters. And there is often ambiguity about which category a question belongs to. These fac- tors make hard for any classifier to reach high accuracy. This datasct has also been used for more constrained learning tasks. For example, Yin et al. 2019 adapts this dataset for zero-shot text classi- fication and Schick and Schütze 2020 uses it for few-shot text classification.
points. Further, for imbalanced datasets like the PC1 and PC2 test sets, AP is a more indicative of performance than accuracy.
4.2 Model comparison
Yahoo
Amazon
Amazon Review Data 2018 (Amazon) Amazon review data 2018 (Ni ct al., 2019) is review rating classification dataset. The dataset comprises over 230 million reviews of 15 million Amazon products along with metadata. We extract product text (e.g. title, description) from the metadata and convert it into a multi-class text classification task. The task aims to classify products into one of the top 23 product categories based product text.
AlMacro
PC1
PC2
K.FC
Selt:
Shots
Figure 2: Absolute improvement in maero average pre- cision (AP) over a BERT 10-shot baseline (gray lines), averaged across unseen classes and three seen/unseen class splits. Note the log scale of the x-axes and the different scales of the y-axes. See Table 1 for details.
PC1 and PC2 Product Classification and 2 (PCI and PC2) are multi-label classification datasets consisting of products sold on an e- commerce website and their associated regulatory classes. Inputs Ij are the concatenation of texts associated with the products, like item names, de- scriptions, and product types. Some classes are general in scope and others are narrow. PC1 has 50 classes, with each sample belonging to at least one class. PC2 can be viewed as a collection of 28 binary classification tasks; one task per class. Labels for both datasets are from human annotators. Each class description comes from policy doc- ument describing the class. These documents much more complex than class descriptions previ- ously studied in the literature. They have multiple sections indicating, for example, specific negative examples for the class and general characteristics of positive examples.
No one approach does best on all four datasets across all amounts of few-shot data, but the match- ing model comes close (Figure 2). The way perfor- mance varies with additional labeled examples is deeee-dde but the matching model exhibits favorable performance in both the low- and high- data regimes. At low sample counts the matching model uses unseen class descriptions to achieve modest levels of performance, whereas the Light- GBM, BERT, and BERT + Adapter baselines tend to struggle. With 10 shots the matching model outperforms these models by an average of 33.8%. 25.4%. and 44.1 %, respectively, emphasizing the importance of class descriptions in enabling classi- fication in zero-shot and k-shot scenarios for small k. With enough labels most baselines eventually learn class boundaries without the aid of descrip- tions: for larger label counts they tend to "catch up" with the matching model. However, the match- ing model remains competitive such cases. This shows that, with a matching model, we may use class descriptions to buoy performance levels in the low-data regime and still make efficient use of extra labels in high-data scenarios.
Metric: macro average precision We measure model performance, relative to a 10-shot BERT baseline, via average precision (AP). To get a sum- mary score, we compute the AP for each unseen class, then take an unweighted average (i.e. macro AP). Next, to obtain relative score, we subtract the macro AP for 10-shot BERT bascline. Finally, we further average across three seen/unseen class splits to reduce the impact of unseen class selection. We chose AP because we do not know priori at which operating point a model will be deployed. AP summarizes performance across all operating
MLPC and SetFit are most competitive when few labels are available, but they struggle improve with more samples. For MLPC this is because la- beled data from unseen classes do not improve its embedding. Instead it uses these data to build class

---

Dataset
Model
shot
10 shots
shots
100 shots
500 shots
1000 shots
LightGBM
8.40
20.07
33,47
$4.57
58.43
BERT
0.00
15.07
40.10
64.08
66.63
Yahoo
Adapter
7.61
35.71
55.98
66.59
67.33
MI.PC
56.88
61.65
61.33
62.57
62.86
SetFit
53.93
59.66
60.77
$7.90
57.49
Matching
46.45
54.17
61.49
63.98
66.56
67.11
LightGBM
28.06
-3.68
10.12
32.59
36.98
BERT
0.00
31.45
36.52
42.75
44.22
Amazon
BERT Adaperr
-28.22
29.07
36.15
44.06
45.72
MLPC
14.86
20.88
20.39
20.94
20.95
SetFit
7.12
28.18
33.25
36.54
3.68
Matching
0.91
24.05
36.01
39.27
44.96
46.96
LightGBM
10.16
11.55
16.82
22.34
BERT
0.00
15.83
19.05
22.68
PCI
Adaper
522669
10.43
17.65
23.18
MLPC
3.92
5.75
6.18
5.48
SetFit
2.67
12.49
14,64
13.18
Matching
-5.51
9.82
16.83
18.67
23.32
LightGBM
-3.82
16.84
24.98
40.22
45.
BERT
0.00
12.22
21.78
39.77
44.22
PC2
Adaper
-1.22
9.60
16.87
36.12
44.37
MLPC
20.47
24.11
24.58
24.90
24.95
SetFit
7.28
17.70
24.76
22.26
32.14
Matching
11.36
13.69
20.09
25.89
40.70
46.75
LightGBM
8.41
11.20
21.35
37.43
BERT
0.00
18.64
29.36
42.32
51.69
Average
Addper
18.63
21.21
31.66
42.49
52.47
MLPC
24.03
28.10
28.12
28.47
36.25
SetFit
17.75
29.51
33.35
32.47
31.11
Matching
13.30
25.43
33.61
36.95
43.88
53.61
Table I:
Average absolute improvement in macro AP (%) compared to 10-shot BERT. We average the scores across
unseen classes and three seen/unseen class splits. The matching model achieves the best performance, on average, across each few-shot setting.
prototypes. If, after model training, the embedding is insufficient to separate positive and negative ex- amples from unseen classes, improving the class prototypes can only help so much. For SetFitt we hypothesize that sentence transformer fine-tuning is best suited to low-data scenarios and has diminish- ing returns with more samples. The other models, however, improve rapidly with additional data.
Dataset
Scen
Unseen
Yahoo
52.83
46.45
Amazon
43.74
0.91
PCI
13.61
-5.51
PC2
53.74
11,36
Table 2: Zero-shot matching model performance (macro AP %) relative 10-shot BERT. average across seen unseen classes and three seen/unseen class splits.
Matching model performance is improved fur- ther by using a more powerful backbone. Using DeBERTa (He et al., 2020) instead of BERT in- creases zero- and max-shot performance by 5.24% and 2.02%, respectively (see Appendix E.2).
Meta-learning To test meta-learning applied matching we follow the same experiment design as above but substitute standard fine-tuning with MAML when training on seen classes. Fine- tuning on few-shot data is unchanged. We use the MAML+ training algorithm, which stabilizes and improves MAML (Antoniou al., 2019), adapted from computer vision models transform- ers as van der Heijden al. 2021. We test the three query set sampling strategies discussed above. See Appendix C for further details.
There is gap between the matching model's seen and unseen class metrics in the zero-shot set- ting, one would expect (Table 2). With enough labels, however, performance unseen classes exceeds the original performance seen classes. This is likely because there are -4 times fewer un- seen classes for the model to learn during few-shot fine-tuning compared to seen class training.

---

Model
shots
shots
50 shots
100 shots
500 shots
1000 shots
Standard training
9.65
0.00
7.16
10.05
16.68
22.06
MAML++ few-shot
8.83
2.26
5.72
8.98
15,81
20.83
Local zero-shot
5.33
0.36
6.40
8.74
15.14
21.14
Global zero-shot
-16.75
-7.45
1.67
5.14
13.76
18.94
Table Average absolute improvement in macro AP (%) compared to 10-shot matching model (BERT backbone) with standard training. We average the scores across unseen classes then across all four datasets.
Local zero-shot query sampling achieves a signif- icant lift over standard training in the zero-shot set- ting (+4.29% on average) with negligible or harm- ful impact in the few-shot setting (Table 3). Hence we recommend this approach for zero-shot but not few-shot learning. Interestingly, we observe improvement for few-shot learning with standard MAML+ perhaps due to the persistent binary classification layer of the matching setting. Global zero-shot query sampling significantly degrades performance in all settings.
mance may vary for other text classification tasks. An inherent disadvantage of the matching formula- tion is that the inference cost scales linearly with both the number of examples and the number of candidate classes, whereas for traditional classi- fier the cost scales with the number of examples. Therefore, matching formulation may be too com- putationally expensive for applications with many classes. Finally, our results may not generalize to much larger or smaller language models as we only considered modest-sized models with 66 to 336 million parameters.
5 Conclusion
Ethics Statement
This paper shows that casting text classification as a matching problem is an effective way to address new classes. Matching models have multiple bene- fits relative other methods in this space. Namely, they (1) leverage both class descriptions and stan- dard labels; and (2) can perform zero- and few-shot classificatinn-bon of which are often needed in practical scenarios. We show that they are able to exploit long, complex class descriptions, achieving 13.30% macro AP with zero-shots (relative to 10- shot BERT) and surpassing few-shot baselines by 22.48% with 10-shots. We also generalize MAML the zero-shot setting and show that proposed method improves zero-shot performance by fur- ther 4.29%. This demonstrates that, when class descriptions are available, the proposed matching models are capable of handling new classes from the beginning (zero-shot, cold-start) to the end of the model development cycle (few- to many-shot).
This research was conducted in accordance with the ACL Ethics Policy. We proposed classifica- tion model that leverages class descriptions which could be the most beneficial to those doing zero- few-shot text classification, especially in the prod- uct compliance space. In our study, we used two internal datasets, but they do not contain any per- sonal or private information, which reduces privacy concerns. It is also important to note that as we used language models pre-trained on internet data (BERT, RoBERTa, DeBERTa, and DistilBERT), model may inherit their biases. However, bias and fairness are not investigated in the paper.
References
Antreas Antoniou, Harri Edwards, and Amos Storkey. 2019. How to train your MAML, In Seventh Interna- tional Conference on Learning Representations.
Limitations
Peyman Bateni, Jarred Barber, Raghav Goyal, Vaden Masrani. Man-Willem van de Meent, Leonid Si- gal, and Frank Wood. 2022. Beyond simple meta- learning: Multi-purpose models for multi-domain, tive and continual few-shot learning. arXiv preprint arX1v:2201.05/11 22151...1.
The proposed model leverages long text (class description) to measure similarity, however text length is limited to 512 tokens due to the underly- ing language model. This limitation may impact the model's performance on longer texts. We used four datasets in our study, but three of them (one public and two internal) are for e-commerce prod- uct classification. As a result, our model's perfor-
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind
Phttps://ww/ aclweb. www.eee.rrrpptta//ooooooo -l-code-ethics

---

Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint 0rXiv:2006.03654
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open- domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1870-1879.
Yutai Hou, Yongkui Lai, Yushan Wu, Wanxiang Che, and Ting Liu. 2021. Few-shot learning for multi- label intent detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 13036-13044.
Jacob Devlin, Ming-War Wei Chang. Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language under- standing, arXiv preprint arXiv: 1810,04805.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficien transfer learning for nip. In- ternational Conference on Machine Learning, pages 2790-2799. PMLR.
Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021. A neural few-shot text classification reality check. In Proceedings of the 16th Confer- ence of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 935- 943.
Guolin Ke, Qi Meng. Thomas Finley, Taifeng Wang. Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu. 2017. Lightgbm: A highly efficient gradient boost- ing decision tree. Advances in neural information processing systems, 30.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic mneaa-lerningg for fast adaptation of deep networks. International conference on ma- chine learning, pages 1126-1135 PMLR.
Omar Khattab and Matei Zaharia. 2020. Colbert: Effi- cient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39- 48.
Tianyu Gao. Adam Fisch, and Danqi Chen. 2021a. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1 1th International Joint Conference on Natu- ral Language Processing (Volume I: Long Papers), pages 3816-3830.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, et al. 2015, Siamese neural networks for one-shot image recognition. In ICML deep learning workshop. volume 2. Lille.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 202 SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing.
Quoc Le and Tomas Mikolov. 2014. Distributed repre- sentations of sentences and documents. In Interna- tional conference on machine learning, pages 1188- 1196. PMLR.
Hao Guo, Xiangyang Li, Lei Zhang, Jia Liu, and Wei Chen. 2021. Label-aware text representation for multi-label text classification. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7728-777777 -7732. [EEE.
Hung-Y Yi Lee, Shang- Wen Li, and Thang Vu. 2022. Meta learning for natural language processing: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 666-684.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith, 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguisticx, pages 8342 8360.
Canjia Li, Andrew Yates, Sean MacAvancy, Ben Hc, and Yingfei Sun. 2020. Parade: Passage representation aggregation for document reranking. arXiv preprint arXiv:2008.0009.
Zied Haj- Yahia, Adrien Sieg, and Léa A Deleris. 2019. Towards unsupervised text classification leveraging experts and word embeddings. In Proceedings of the 57th annual meeting of the Association for Computa- tional Linguistics, pages 371 1779.
Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. 2021 Pretrained transformers for text ranking: Bert and beyond. Synthesis Lectures Human Language Technologies, 14(4):1-325.
Kishaloy Halder, Alan Akbik, Josip Krapac, and Roland Vollgraf. 2020. Task-aware representation of sen- tences for generic text classification. In Proceedings of the 28th International Conference on Computa- tional Linguistics, pages 3202-3213.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint 1907.1

---

Sean MacAvaney, Andrew Yates, Arman Cohan, and
Natraj Raman, Sameena Shah, and Manuela Veloso. 2022. Structure and semantics preserving document representations. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 780- 790.
Nazli Goharian. 2019. CEDR: Contextualized em-
beddings for document ranking. In Proceedings of the 42nd international ACM SIGIR conference on research and development in information retrieval, pages 1101-1104.
Yu Meng, Yunyi Zhang, Jiaxin Huang, Chenyan Xiong, Heng Ji, Chao Zhang, and Jiawei Han. 2020. Text classification using label names only: A language model self-training approach. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 006-9017.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert Sentence embeddings using siamese werrsn In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Associa- tion for Computational Linguisties.
Anthony Rios and Ramakanth Kavuluru. 2018. Few- shot and zero-shot multi-label learning for structured label spaces. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical Methods in Natural Lan- guage Processing, volume 2018, page 3132. NIH Public Access.
Jinseok Nam. 2019. Learning label structures with neu- ral networks for multi-label classification.
Jinseok Nam, Eneldo Loza Meneía, and Johannes
Fürnkranz. 2016. All-in text: Learning document, label, and word representations jointly. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 30.
Stephen Robertson, Hugo Zaragoza, et al. 2009. The probabilistic relevance framework: BM25 and be- yond. Foundations and Trends in Information Re- trieval, 3(4):333-388)
Nihal Nayak and Stephen Bach. 2020. Zero- shot learning with common sense knowledge graphs. arXiv preprint arXiv:2006.1000 10713.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv: 191001100000108.
Jian Ni, Jiacheng Li, and Julian McAuley. 2019. Jus- tifying recommendations using distantly- labeled re- views and fine- -raing aspects. In Proceedings of the 2019 conference empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP- (JCNLP), pages 188-197.
Timo Schick and Hinrich Schütze. 2020. Exploit- ing cloze questions for few shot text elassification and natural language inference. arXiv preprint arXiv:2001.072 0.6676..76....
Nikolaos Pappas and James Henderson. 2019. Gile: A generalized input-label label embedding for text classifica- tion. Transactions of the Association for Computa- rional Linguistics, 7:139-155.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352.
Hyunji Park, Yogarshi Vyas, and Kashif Shah. 2022. Efficient classification of long documents using trans- formers. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume Short Papers), pages 702-709.
Tim Schopf, Daniel Braun, and Florian Matthes. 2022. Evaluating unsupervised text classification: zero-shot and similarity-bused approaches. In Proceedings of the 6th International Conference on Natural Lan- guage Processing and Information Retrieval, 2022, pages 6-15.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021, True few-shot learning with language models. Ad- vances in neural information processing systems, 34:11054 11110001-110...... 11070.
Konstantinos Sechidis, Grigorios Tsoumakas, and loan- nis Vlahavas. 2011. On the stratification of multi- label data. In Machine Learning and Knowledge Dis- covery in Databases: Earopean Conference, ECML PKDD 2011, Athens, Greece, September 5-9, 2011, Proceedings, Part III 22, pages 145-158. Springer.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. 2020a. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.0m 000447.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho. and Iryna Gurevych. 2020b. Adapterhub: A framework for adapting transformers. arXiv preprint arXiv:2007.07779
Wissam Siblini, Mohamed Challal, and Charlotte Pasqual. 2020. Delaying interaction layers in transformer-basee encoders for efficient open domain question answering. arXiv preprint arXiv:2010.08422.
Pushpankar Kumar Pushp and Muktabh Mayank Sri- vastava. 2017. Train once, test anywhere: Zero- shot learning for text classification, arXiv preprint arXiv:1712.00972.
Jake Snell, Kevin Swersky, and Richard Zemel. 2017. Prototypical networks for few-shot learning- Ad- vances in neural information processing systems, 30.

---

Mohamed Trabelsi. Zhiyu Chen, Brian D Davison, and Jeff Heflin. 2021. Neural ranking models for document retrieval. Information Retrieval Journal, 24:400-444.
Pengcheng Yang. Xu Sun, Wei Li, Shuming Ma, Wei Wu, and Houfeng Wang. 2018. Sgm: Sequence gen- eration model for multi-label classification. In Pro- ceedings of the 27th International Conference on Computational Linguistics, pages 3915-3926.
Lewis Tunstall, Nils Reimers, Unso Eun Sco Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts. arXiv preprint arXiv:2209. 1055..
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoe V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems. 32.
Niels van der Heijden, Helen Yannakoudakis, Pushkar Mishra, and Ekaterina Shutova. 2021. Multilingual and cross-lingual document classification; A meta- learning approach. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguisties: Main Volume, pages 1966-1976.
Chenchen Ye, Linhai Zhang. Yulan He. Deyu Zhou, and Jie Wu. 2021. Beyond text: Incorporating metadata and label structure for multi-label document classifi- cation using heterogeneous graphs. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3162-3171.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap.
Daan Wierstra, et al. 2016. Matching networks for one shot learning. Advances in neural information processing systems, 29.
Zhiquan Ye, Yuxia Geng, Jiaoyan Chen, Jingmin Chen, Xiaoxiao Xu, SuHang Zheng, Feng Wang. Jun Zhang, and Huajun Chen. 2020. Zero-shot text classification via reinforced self-training. In Proceedings of the 58th Annual Meeting of the Association for Compu- tational Linguistics, pages 3014-3024.
Marilyn Walker, Heng Ji, and Amanda Stent. 2018. Pro- ceedings of the 2018 conference of the north ameri- can chapter of the association for computational lin- guisties: Human language technologies, volume I (long papers). In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume (Long Papers).
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNI.P-ICCNIP)) P). pages 3914-3923.
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. arXiv preprint arXiv: 104. 14690.
Jingtao Zhan, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2020. An analysis of BERT in doc- ument ranking. In Proceedings of the 43rd Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1941- 1944.
Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL HLT 2018, pages 1112-111 1122. Association for Computational Linguis- tics (ACL).
Jingging Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. 2019. Integrating semantic knowledge to tackle zero-shot text classification. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. Volume 1 (Long and Short Papers), pages 1031-1040.
Lin Xiao, Xin Huang, Boli Chen, and Liping Jing. 2019. Label-specific document representation for multi- label text classification. In Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNIP--JCCNIP). pages 466-475.
Min-Ling Zhang and Zhi-Hus Zhou. 2013. review on multi-labet learning algorithms. IEEE transactions on knowledge and data engineering. 1818):1899 1837.
Yumo Xu and Mirella Lapata. 2020. Coarse-to-fine query focused multi-documedt summarization. In Proceedings of the 2020 Conference on empirical methods in natural languuge processing (EMNLP), pages 3632-3645.
Wenjie Zhang, Junchi Yan, Xiangfeng Wang, and Hongyuan Zha. 2018 Deep extreme lubti-label learning. In Proceedings of the 2018 ACM on inter- national conference on multimedia retrieval, pages 100-107.
Guangxu Xun, Kishlay Jha. Jianhui Sun, and Aidong Zhang. 2020. Correlation networks for extreme multi-label text classification. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1074- 1082.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.

---

A Additional related work
ding transformation ge : X Id with trainable parameters 0 which maps samples from the text- based instance space to the embedding space. To handle text input we follow Dopierre et al. 2021 and implement this map with a transformer (BERT (Devlin et al., 2018)).
Information retrieval The goal for an informa- tion retrieval (IR) system is to return a large pool of seadiidatsswwiche are most rele- vant to given query. We will assume the objects to be text-based documents. We may view classi- fication with class descriptions as an IR problem; the example text provides the query and the class descriptions are the documents to be searched. IR systems are often used as components of other sys- tems: QA (Chen et al., 2017), summarization (Xu and Lapata, 2020), recommendation (Walker et al., 2018), and search and navigation (Raman et al., 2022). Modern architectures can generally be bro- ken into two stages: fast, high-recal retrieval stage (e.g. BM25 (Robertson al., 2009)) seek- ing to quickly pare down the list of possibly rele- vant documents, and a slower but more accurate ranker which ranks the pruned list of documents by relevance to the query. Many document ranking models compute a score for each document with respect to the query. much like matching model. Indeed, neural language models like BERT (De- vlin al., 2018) are common choices for ranking models (Zhan et al., 2020; MacAvaney al., 2019; Lin et al., 2021; Li et al., 2020). One drawback of such architectures, however, is the difficulty with which they scale to longer documents. Researchers have proposed various techniques (Lin et al., 2021) for ingesting longer input texts such as aggregating scores across different document sections (MacA- vaney et al., 2019; Park et al., 2022) and using net- works with larger maximum token lengths (Yang al., 2019). Though recent work has begun to question the effectiveness of such methods for text classification (Park et al., 2022).
from
Episodic training To simulate the few-shot test- ing environment (few labels per class), train the transformer go with series of few-shot episodes. In each episode, we sample support sets Sj.0 SJ,1. and a query set Tí from triplet training dataset Derain (Zz,, Cj, Vij]] for each class j. where = fl....,Ny .N3 and J = {1....,P] . are the in- dex sets of text samples and classes. From the samples (zi, cj. yij) of the support sets, com- pute class prototypes Cj, and C₁ for class j with the embedding transformation ge. and negative and positive text samples in Sj.0. SJ,1, respectively.
CJAb = 1S
Σ g0(x), xCSS,u
JE c 10,
For every text sample æ in query set Tj. we com- pute the Euclidean distances between its embed- ding x = (2: and class prototypes Cj.0 and Cj.1 and then we define the output distribution of proto- typical classifier given an input as
[mx,e)) = Pr(mi = 1|,, cj) exp( d(x, cj,1))
(2)
c,,0)) ) + (xp(-d(x,c,..)
d(x,
where d(x, c) - |x c||2 is the Euclidean dis- tance. The output distribution is essentially a soft- max over the distances between embedding x and class prototypes Cj.0 and Cj,1 Given a labeled ex- ample (I. Cj. Vi.j k). k E {0, 1} from query set Tj. the sample cross-entropy loss between ground truth label yt,j and prediction f(z,, c,) is
B Multi-label prototypical classifier
Our multi-label prototypical classifier (MLPC) adapts prototypical networks (Snell et al., 2017) from multi-class image classification to the multi- label text classification setting. In essence, MLPC uses examples from seen classes to learn a met- ric space in which it can classify new examples via comparison against seen and unseen class pro- totypes. Each class has two prototypes- one for positive (in-class) examples and one for negative (out-of-class) Seamples-frmedd formed by averaging the embeddings of few-shot positive and negative in- stances of each class. Hence our objective is to learn a dddimensional metric space with an embed-
CE Loss = - log log(Pr( = kl
d(x, d(xx log Σ exp exp(-d(x,, exp(-d(x,cGx)))
(3)
We learn via SGD by minimizing the cross- entropy loss summed over labeled examples, re- peating the process multiple times for each class with randomly generated support and query sets. See Algorithm in Appendix C for pseudocode.

---

Inference The datasets can be expressed in triplet format Dist { Cj. 9ky) Tuu Just where lest and Jeett are the index sets of test sam- ples and test classes. The classifier produces proba- bility score fiz Gÿ) of 2; belonging to class j for all (i.j) E Lust x Juest given few-shot support sets Sy., J j E Jees
C.2 Multi-label prototypical classifier
We used the following hyperparameter configura- tion to learn the parameters of embedding transfor- mation via episodic training with prototypical network:
Base model: berf-base-uncased
cz) Pr(yy.j 1[z;; exp(-d(x; C2,1))
Dropout: 0.1
Batch size: 64
exp(-d(x;, c3,0)) + (xxp((d((x:))) Cz,1))
v(i.j) E Lust x Just
Episodes: 5000
(4)
Note that MLPC does not use the class descrip- tions, only the few-shot samples. We found that incorporating class descriptions in (Hou et al.. 2021) (using them to anchor class prototypes) hurt performance.
Maximum learning rate: {1e-6, 2e-6, 5c-6, 1e-5, 2e-5)
Learning rate schedule: linear warmup to max- imum value 6% of training steps and then linear decay
C
Model training
Early-stopping: True
This section provides details of how each model was trained, including model backbones and hyper- parameter choices.
Our experiments show that few-shot Yahoo and PCI datasets favor lower learning rates (best at le- 6) while we get the best performance on the PC2 dataset at le-5.
C.1 Matching model
We used the following hyperparameter configu- ration when training the zero-shot version of the matching model across all datasets:
Algorithm gives an overview of the episodic training for MLPC.
C.3 LightGBM
Base model: bert-base-uncased
For LightGBM we used the default model param- eters with objective= 'binrr''' for the PC1 and PC2 datasets and wewectiveemulticllae the Yahoo dataset. We trained the model on the few-shot datasets:
Dropout: 0.1
for
Epochs: 2
Batch size: 32
Learning rate schedule: linear warmup for 10% of an epoch to maximum value of 5c-6, then linear decay to 0
Yahoo train single multi-class model using samples in the few-shot dataset (rather than training separate binary classifiers for each class then taking the argmax of probability scores)
When tuning the zero-shot matching models on few-shot data we changed the following hyperpa- rameters:
PC1 train a separate binary model for each unseen class using all samples in the few-shot dataset
Epochs:
Learning rate schedule: start at 5e-6 with co- sine decay to 0
PC2 train separate binary model for each unseen class using all samples in the few-shot dataset
For the few-shot experiments randomly lected 5% of the Yahoo training set use vali- dation data and 1 5% of the Amazon, PC1, and PC2 training sets. For the sanity check experiment on the Yahoo dataset we used the validation set given in (Yin et al., 2019).
We applied the same text preprocessing for the other models and used a simple Bag-of- Words text fcaturization. The PC1 and PC2 datasets each have four categorical features associated with them,

---

Algorithm 1: Episodic training of MLPC. E denotes the number of training episodes, Ns and Ny denote the number of support set and query set, respectively. RandomSample(D. D, Vij = k, N) denotes uniformly sampled subset of D with samples satisfying the condition of Vij k.
C.5 BERT Adapter
We followed the adapter framework proposed Pfeiffer et al. 2020b to insert adapter module with a 768 x 48 down-projection layer and a 48 x 768 up- projection layer in every BERT transformer block. We then train BERT Adapter models using the following hyperparameters:
Base model:
Data: Triplet training data
Duain = {(z,, Cjn II5]
Epochs: 10
Initialize 9o
// with an off-the-shelf
Learning rate schedule: linear warmup for 10% of an epoch to maximum value of 5c-5, le-4, or 2e-4, then linear decay 0
pre-trained transformer for e in (1,.... F.) do for j in J do Obtain slice of dataset
Dropout: 0.1
Dj {Ii ej Bi))ii for class çj
We have found that training a BERT + Adapter model with a higher maximum learning rate works better since the adapter weights are initialized ran- domly without pre-training.
S30 =
RandomSample( Dj. Vij 0, Ns)
SJ.1 -
RandomSample( Dj, ", Mhj 1, Ns)) Т RandomSample( Dj S,,0*
C.6 SetFit
1 Dij = or 1, Nr) Compute {c, C2,0 from S3o, S₁,1
We used the official SetFit package² to train the SetFiit models. Wherever possible, we chose de- fault recommended parameters. We list these parameters below.
Compute loss L from T3 and
{cj.0. C5,0}
10
Update 0 < 0 V
Sentence transformer backbone: all-mpnet-base-v2 We chose this pre-trained backbone because it gave the best all-around performance of the available sentence transformers
// iterate through each class
// iterate through episodes
three of which we converted to text and appended to the example text: product type, minimum age, and maximum age. The LightGBM model, however, can ingest categorical features directly. Therefore we also passed these four features to LightGBM as categorical variables using an ordinal encoding.
Contrastive loss:
cosine similarity
Epochs: for contrastive training, 25 for tun- ing the classification head
Learning rates: 2e-5 for contrastive training, 1e-2 for tuning the classification head
C.4 BERT
We froze the sentence transformer while train- ing the classification head as otherwise we faced CUDA OOM issues.
We trained the BERT models text data with the same preprocessing steps as with the matching model and MIPC, using the following hyperparam- eters:
The number of iterations (text pairs to gener- ate for contrastive lcarning) differed based on the number of few-shot samples kt Using a fixed number of iterations across all values of k results in an unwieldy number of samples for large k. We set the number of iterations to: 5 if k: 500, 2 if k: 500, and if k: 1000.
Base model: bert-base-uncased
Epochs: 10
Learning rate schedule: linear warmup for 10% of an epoch maximum value of le-6, then linear decay to
'https://github cow/huggingface//eeeee htt //www. sbert. net http///////eeeraieedmeeeeeeeee html#senterd embedding- models/
Dropout: 0.1

---

We trained separate binary (one-vs-all) model for each unseen class on each few-shot dataset be- fore evaluating on the test set. We used the same text processing steps as for the other models.
Slightly different hyperparametters (shown be- low) worked best for each query sampling method. We selected the hyperparameters based on perfor- mance on loss on the validation set.
C.7 Meta-learning
MAML++: 60 evaluation tasks
The training and evaluation process for the meta- learning experiments closely followed that for the matching model. We simply substituted the stan- dard method of training the zero-shot model with MAML++ (Antoniou et al., 2019) training routine. The model inputs, few-shot fine-tuning process, and evaluation are all unchanged. We used a BERT model architecture.
Local zero-shot query sampling: inner and outer loop learning rates were le-5
Global zero-shot query sampling: 50% of classes reserved for the query
D Few-shot datasets
In this section we provide additional details about the Yahoo, PC1, and PC2 few-shot datasets. We repeated the procedures below five times, selecting different seen and unseen classes in each case to test the models' robustness. However in this paper only report results for the first three splits: 0, 1, and 2.
Meta-training consists of fixed number of epochs, each with the same number of training episodes. At the end of each epoch we estimate the loss on the validation with a fixed number of episodes. We use the model checkpoint with the lowest validation loss. When drawing samples for the support and query sets we balance the number of positive and negative examples. The match- ing formulation naturally induces imbalance between positives and negatives for datasets with complete label information and large numbers of classes (all datasets but PC2). Without enforcing this balance the model sometimes becomes trapped local minima; assigning every sample score close to Samples for the support and query sets are drawn with replacement.
D.1 Yahoo
To align the Yahoo Answers dataset from Pushp and Srivastava 2017; Gururangan al. 2020 with our few-shot experiments, we pool all the exam- ples from the original train, test, and dev sets and build our own train, few-shot, and test sets. We first randomly select two of the ten classes to act as unseen classes (classes and 4 for the experi- ments that follow). From each of the remaining eight seen classes then sample 80% of the ex- amples to serve as Dusin. with the other 20% going to Drest- We randomly select 1000 examples from each of the two unseen classes to build Drew-shot- We allocate the remaining examples from the un- seen classes to Drest. In order to ease the compu- tational burden of evaluating on the test set, we further downsample Drest to form Dreat-amall and evaluate that set instead. Finally, we convert D Diew-shot and Diest into triplet format so that each example (zz, 31) E D mapped to p examples {(x;, Cj, 12ys = }))} and each exam- ple (x,, y/) E Diew-shot U Drest-small is mapped to p + g = 10 examples {(x,, Cj. 1l [ys = } See D for details about the datasets. We use the same class descriptions described in Section E. We measure performance by averaging the AP of the model across unseen classes, i.e. compute AP score for each unseen class, then take the unweighted average as the final score.
Below we give the important hyperparameters from the meta-learning experiments.
Batch size:
Max token length: 512
Minimum learning rate: 1e-8
Classes per task:
Support samples per class: 3
Query samples per class:
Inner and outer loop learning rates: 5e-5
Total epochs: 50
Episodes per epoch: 200
Multi-step loss epochs: 10
Number of evaluation tasks during validation:
Table 4 shows the distribution of samples across the different splits.
200

---

Split
Total samples
Samples per seen class
Samples per unseen class
Train
934.4k
116.8k
Test
523.6k
29.2k
145k
Test (small)
100k
6k
26k
n-shot
2n
0
Table 4: Number of samples in the Yahoo few-shot data splits. n E {10, 50, 100, 500, 1000).
D.2 Amazon
pool of examples falling into any of the 10 classes." From this pool we sample 500 examples from each class to form Dow-shott We do not have enough samples for 1000-shot learning. Note that because examples can belong to multiple classes, it is pos- sible for Diew-shot to contain more than 500 sam- ples from a given class. We assign the remaining examples from unseen classes Diest We then split the data from seen classes between Duain and Diest at an 80/20 ratio, using iterative stratification (Sechidis et al., 2011) 1) to ensure different label com- binations are distributed evenly across the train and test sets, In the end Duain = 29,91 917 and D = 13, 782. Finally, we convert the splits into triplet format.
We build multi-class e-commerce product classi- fication dataset by extracting the metadata from Ni al. 2019. We take 'asin' as the unique identifier for product examples, 'category" as the target class, and the concatenation of 'title", 'description', and 'feature' as example text. To build a balanced classi- fication dataset, we first choose the top-23 product categories which have more than 4000 product ex- amples. We then downsample and obtain 4000 product examples per class. For each class, need class description which describes the product category. We manually search on Wikipedia and Google and combine text from difference sources with minor modifications, e.g. removing irrelevant information like citations, pronunciation, root, etc.
For compatibility with MLPC, Diew must also have 500 positive examples from each seen class. We sample these from Drrains upsampling for classes with fewer than 500 positive examples.
Next, we construct Drain, Diew-shot. and Drest by selecting 5 unseen classes and 18 seen classes. For each seen class, we put 2000 examples in Dtrain- 1000 examples in Diew-shol- and 1000 examples in Diesti while for each unseen classes, we only put 1000 examples in Drew-shot and 1000 examples in Drest- Table 5 shows the number of positive examples for each class.
Table 6 shows the number of positive examples for each class in Drrai and D
D.4 Product Classification (PC2)
Unlike with the PCI dataset, here the labels are sparse. With the PC1 data had full knowl- edge of the multi-label for each example, e.g. y PC1 (0, 1, O, 0, 1, .)) L.., but with the PC2
D.3 Product Classification 1 (PC1)
data we have only partial knowledge, e.g. v, pP2 = 1,,,0.
Product Classification 1 (PC1) dataset contains 50 classes which have between 57 and 4,952 exam- ples with mean of 1,319. Examples can belong to multiple class (multi-label classification problem) with an average of 1.35 classes and a maximum of seven. Labels were assigned by manual review- ers. The labels are dense in the sense that every example is positive example for some classes and a negative example for all the others. As a result, after we convert these multi-label datasets into their triplet versions, there are many more negative ex- amples than positive examples; the modal example becomes one positive and 49 negative examples.
It is still possible to con- vert such labels into a triplet format, however: (I,, of PC2) is mapped to {(zi, (Ii, 3j PC2) < < p. yig PC2 }?} rather than {(x,, Cj. Miy p} (replacing p with p + q for the test set). For the PC2 dataset we perform this conversion before constructing the train, few-shot, and test sets.
PC2) [ S <
The train set D Drrain has, wherever possible, 50,000 positive and 50,000 negative samples from each seen class and samples from unseen classes. Therefore the PC2 train set is much more balanced than the PCI one. of 22 seen classes have fewer
*We only consider classes with at least 700 examples as viable candidates for unseen classes to ensure there are enough to split between Drex-sut and Dust
We construct Durin. Diew-shot and Drest by first selecting = 10 unseen classes (20%), building a

---

Split
Total samples
Samples per seen class
Samples per unseen class
Train
36,000
2,000
0
Test
23,000
1,000
1,000
n-shot 2n
Table 5: Number of samples in the Amazon few-shot data splits, where n E {10, 50, 100, 500, 1000 y. In each split, there are unseen classes and seen classes. For split O, the unseen classes are: Home Kitchen; Tools and Home Improvement; CDs and Vinyl; Movies and TV: Musical Instruments, while the seen classes Shoes and Jewelry; Books; Automotive; Sports and Outdoors; Electronics; Toys and Games; Cell Phones and Accessories; Kindle Store; Arts, Crafts and Sewing: Grocery and Gourmet Food; Patio, Lawn and Garden; Office Products; Pet Supplies, Industrial and Scientifie; Video Games; Appliances; Software; Collectibles and Fine Art.
Clothing,
than 100,000 samples due lack of labels (with a minimum of 7,223). We equip Drow-shot with 1, 1,000 positive and 1,000 negative samples. When we discuss k-shot learning with respect to the PC2 dataset, we use k positive and & negative samples. As with the PCI dataset, the few-shot data are nested. The test set Diest consists of 1,000 positive examples and 19,000 negative examples for each class.5 We intentionally created an imbalanced test set to mimic production environment, in which examples from any given class are expected to be rare.
E
Additional experiments
E.1 A sanity check
Aiming to verify the correctness of our implemen- tations of the methods from 3 we conducted san- ity check by replicating an experiment from (Yin et al., 2019) using our models and comparing the results. The experiment involves zero-shot multi- class text classification using class descriptions. Our multi-label prototypical classifier not a zero- shot model, as it requires few-shot support sam- ples for each class. In this experiment, we take 1000 support samples from the dev set (separated from train and test sets). The comparison between multi-label prototypical classifier and other zero- shot models is not fair; however, this experiment can still show us whether our implementations are reasonable.
Table gives an overview of the number sam- ples (counting both positive and negative) for each of the classes in the triplet versions of the PC2 datascts.
D.5 Dataset difficulty
E.1.1 Setup
All models perform better the PC1 and Ama- zon datasets compared Yahoo and PC2 (Table 1). This is likely because the Yahoo and PC2 prob- lems are harder than PCI. The labels in the Yahoo dataset are noisy. They were assigned by users of Yahoo! Answers and are often wrong. Further- more, it is common for questions to plausibly fall under multiple categories even though the labels are multi-class. Section E.1 shows that our results are in line with those previously published in (Yin al., 2019). The PC2 dataset is challenging by de- sign. Its list of classes is not exhaustive, but rather consists product classes which proved especially difficult for existing ML models. Therefore we expect lower scores on this dataset at small label counts. However, most models appear to scale well on this dataset, suggesting that high performance is achievable with more labels.
For this experiment we use Yahoo Answers dataset introduced in Zhang et al. 2015, with the data split and experiment setup used in Yin et al. 2019. The objective for this dataset is to use the concatenation of a Yahoo Answers question and its top answer to predict which of 10 categories the poster filed the question under: Society & Cul- ture, Science Mathematics, Health, Education & Reference, Computers & Internet, Sports, Busi- ness & Finance, Entertainment & Music, Family & Relationships, Politics & Government.
We use the class descriptions (Yin et al., 2019) found worked best for this dataset: cj "this text describes something about wy"" where my is the 1-3 word name of the category (e.g. "Sports"). The dev and test sets contain 6k and 10k examples from each of the 10 classes, respectively (60k and 100k samples total). There are two non-overlapping train sets, each with a different set of five seen classes with 130k samples per class. The class
*In the PC2 test set two classes have only 9,000 negative examples und one has 4,000 due to data availability.

---

Class
Seen
Train samples
Test samples
0
100,000
20,000
100,000
20,000
0
20,000
100,000
20,000
100,000
20,000
Class
Seen
Train samples (+)
Test samples (+)
O
10,000
100,000
20,000
315
48,790
20,000
867
226
100,000
20,000
192
63.334
20,000
3
100
419
100,000
20,000
847
216
19,842
20,000
5
829
207
20,000
499
126
100,000
20,000
7
S55A55 SET
793
206
42,009
20,000
8
890
223
100,000
20,000
9
1,071
280
O
10,000
10
100
1,393
7,223
5,000
11
760
438
24,253
20,000
12
1,026
263
19
O
20,000
13
853
219
20
100,000
20,000
14
1,323
364
21
O
20,000
15
1,756
469
22
100,000
20,000
16
1613
449
23
25,899
20,000)
17
107
714
24
V
100,000
20,000
18
1,241
528
25
V
100,000
20,000
19
1,503
554
26
V
81,255
20,000
20
810
202
27
V
15,885
20,000
21
388
111
22
792
199
Table Summary of the triplet versions of the PC2 dataset (split 0).
23
2,676
1,569
24
791
198
25
818
218
26
986
246
27
522
130
28
2,371
1,089
index sets for the two train sets are (1,3, 7,99 and {2, 4, 6, 8, 10), respectively. Converting these datasets to from multi-class format a triplet format increases the size of the dev and test sets by factor of 10 and the train set by a factor of 5. Each example is mapped to positive example for one class and negative examples for the remaining classes. The presence of unseen classes in the vali- dation set means that this is not a "true" zero-shot problem.
29
1,026
579
30
759
190
31
1,107
305
32
860
218
33
100
437
34
100
454
35
54
36
222
37
672
38
286
39
669
40
268
990
250
42
382
101
We compare our models against two models pro- posed in (Yin al., 2019). The "BiRT" model is equivalent to our matching model, but uses max token length of 128 (as opposed to 512 for the matching model). The "MNLI-ennaallmen" model, ex-
43
45
11
44
100
965
45
227
567
46
822
208
47
100
416
48
1,332
350
model is the same as the Binary-BEii cept it is pre-trained on MNLI All models are fine-tuned on the training data, with hyperparam- eters tuned on the dev set, then evaluated on the test set. We measure performance via multi-class accuracy, as in (Yin al., 2019). The Binary- BERT, MNLI-entailment, and matching models use an additional hyperparameter which dampens the matching scores for seen classes by a fixed amount because others (Pushp and Srivastava. 2017: Guo
49
100
476
Table 6: Summary of the (non-triplet) versions of the PC1 dataset (split 0). We report the number of positive samples for each class.

---

al., 2021) have noted that matching scores for un- seen classes tend to be lower than for seen classes in the zero-shot setting.
Increasing the size of the backbone generally leads improved performance. (see Figure 3 and Table 10). The larger DeBERTa and BERT- large models consistently outperform RoBERTa and BERT-base, which in turn tend to outperform DistilBERT. Despite BERT-large having almost twice as many parameters as DeBERTa, both mod- els have comparable performance. This shows the benefit of improved pre-training strategies.
E.1.2 Results
The results in Table 8 confirm that our implemen- tations are sound. We find that similar or better performance on unseen classes when comparing our approaches against those in Yin et al. 2019. approach improves significantly on the accuracy scores from Yin et al. 2019. This is not surpris- ing given that MLPC gets to use few-shot samples from unseen classes, whereas the other models are zero-shot.
able to get
unseen classes our MLPC
The downside to larger models is their increased training and inference cost. The train and inference times for the backbones are proportional to the parameter counts (Table 11). DeBERTa and BERT- large are about 1.7 and 2.6 times slower than BERT, respectively, while DistilBERT takes about 0.6 the time of BERT. Which backbone is appropriate will depend on the accuracy and compute constraints of the problem at hand, but DeBERTa is a good all- around choice.
E.2 Matching model backbones
While we used BERT-base as the backbone for the matching model the main paper, other language models could lead to different levels of perfor- mance. In this experiment we replaced BERT-base with other transformer models with sizes ranging from 66 million to 336 million parameters (Table 9). The model weights came from Hugging Face: We replicated the experiments from the main pa- per on seen/unseen class split 0, changing only the backbone models. To understand the trade-offs between compute cost and performance we also tracked the training time and inference latency for each backbone (Table 11 1).
E.3 Zero-shot comparison against unsupervised similarity- -based approach
In this section, we compare the matching model and LLi2TrransormerVee (Schopf et al., 2022), unsupervised similarity-based approach. LbI2Trrasformer/ operates by relying on man- ually defined keywords for each class, and em- ploys pre-trained embedding transformation, such as Doc2Vec (Le and Mikolov, 2014), S- BERT (Reimers and Gurevych, 2019), or Sim- CSE (Gao et al., 2021b). The transformation em- beds keywords, example texts, and classes into the same space. Notably, the class representa- tion is constructed the average embedding of set of most relevant examples given key- words. In our experiment, we observe stronger results with S-BERT among the three embedding transformations. To perform zero-shot classifi- cation, Lb12Transformer leverages class key- words, while the matching model uses class de- scriptions. Lb12Transformerve. is aimed at zero- shot learning, and lacks clear way enhance performance with additional few-shot labels, we focus on the zero-shot setting in this experiment.
Yahoo
Amazon
<80
A(Macro AP)
430
PC1
PC2
EETTbae 3104
Shots
Figure 3: A comparison of different matching model backbones. The baseline in this case is a matching model with BERT backbone trained 10 few-shot samples for each class (gray line). Note the log scale the x-axes and the different scales of the y-axes. See Table 10 for details.
The matching model significantly outper- forms Lbl2Transformervec across all four datasets (Table 12). The lower performance of Lb12TransformerVee can be attributed, in part, to the process of extracting class keywords from long class descriptions. These deseriptions often involve complex elements such as negations and
"https: httg://huugggggccccccooc co/ used the following models: bert-base-uncased. roberta-bar base, mi wattrr//teeeeetaa--aabae and pert-largg-uncase.
In par-
ticular. we

---

Split
Split
Model
Seen classes
Unseen classes
Seen classes
Unseen classes
Binary-Bu BERT
Binary-BERT
72.60%
44.30%
80.60%
34.90%
MNLI-entailmen
70.90%
50%
77.30%
45.30%
MLPC
71.41%
58.21 %
77.11%
57.67 %
Matching
66.54%
56.50%
74.84%
44.52%
Table Comparison models with entailment approach on the original Yahoo Answers dataset (zero shots). We measure multi- clasi-class accuracy. We obtained metrics for the first two models from Yin et al. 2019.
Model
Parameters
Paper
F Future work
DistilBERT
66 million
Sanh et al. 2019
There are myriad avenues for building upon the work described in this paper. Pre-training the matching model either entailment clas- sification dataset could improve performance sig- nificantly. For instance one might consider pre- training on one or more of the datasets in this pa- per, then fine-tuning on the other. The zero-shot capabilities of the matching model create the possi- bilities of employing active learning to select few- shot samples pscudo-labeling to bootstrap addi- tional training samples. It would also be interesting to investigate how the models scale even more labeled samples. The class descriptions consid- ered in this work were, for the most part, short enough to fit within the maximum token length of the transformer-baat based models. Additional work is needed to generalize these approaches to even longer class descriptions. Analyzing the attention layers or token salience scores could provide valu- able insights into what information matching mod- els are using to make predictions. There are also other ways one might exploit the structure inherent in policy documents besides separating them [SEP) tokens (Li et al., 2020). Finally, there are "free" means of further improving model performance, which we did not explore in this work. These in- clude, but are not limited to, applying domain adap- tive and/or task adaptive pre- mre-training (Gururangan et al., 2020), performing additional hyperparam- eter optimization, or incorporating additional fea- tures beyond text (categorical, relevance features from simple retrieval models like BM25 (Robert- son et al., 2009), etc.) (Siblini et al., 2020; Chen et al., 2017).
BERT-base
110 million
Devlin al. 2018
RoBERTa
123 million
Liu al. 2019
DeBERTa
184 million
He 2020
BERT-large
336 million
Devlin 2018
Table 9: Transformer backbones tested for the matching model.
keywords or keyphrases to be excluded from the class. Such information cannot be fully captured by small number of keywords, potentially leading mostly
suboptimal results. Furthermore, relied default values for Lbi2Transformer hyperparameters (besides keyword extraction and the pre-trained embedding transformation). It is likely that hyperparameter optimization would improve the metrics reported here.
E.4 Full meta-learning results
This section reports the full results of the meta- learning experiments from Section 3.2.1. Figure shows a summary plot and Table 13 provides the raw numbers.
Yahoo
Amazon
A.Macro AP)
PC1
PC2
430
Shots
Figure 4: Absolute improvement in macro AP of meta- learning approaches over 10-shot matching model with a BERT backbone (gray lines), averaged across unseen classes. Note the log scale of the x-axes and the different scales of the y-axes. See Table 13 for details.
G Example inputs
Here we give examples of the class descriptions and example texts fed into our models, after prepro- cessing. Note that only the matching model used the class descriptions.

---

Dataset
Backbone
shot
10 shots
shots
100 shots
500 shots
1000 shots
DistilBERT
-7.58
0.63
5.85
9.39
14.97
16.42
BERT-base
-7.17
0.00
7.66
11.16
15.27
16,03
Yahoo
RoBERTa
-8.97
-1.35
5.37
9.87
16.03
17.10
DeBERTa
-1.83
3.76
8.30
12.19
16.82
17.89
BERT-large
-3.43
2.65
9.85
12.36
16.18
16.43
DistiBBERT
-34.20
-00.75
2.43
7.24
18.03
22.57
BERT-hase
-17.01
0.00
12.24
15.58
22.46
25.11
Amazon
RoBERTa
-18.90
-6.65
7.60
10.80
20.68
23.52
DeBERTa
-5.92
4.59
15.74
19.25
25.15
26.67
BERT-large
-12.27
7.53
14.55
16.15
23.85
25.53
DistilBERT
-19.12
-3.24
3.06
4.65
8.30
BERT-base
-13.48
0.00
4.04
5.72
9.18
PC1
RoBERTa
-11.99
1.14
4.92
6.02
9.02
DeBERTa
-11.02
1.19
6.06
7.12
9.51
BERT-large
-7.10
2.46
5.23
6.87
9.46
DistiBBERT
-3.11
-1.98
1.41
4.87
17.89
23.58
BERT-hase
-1.24
0.00
3.99
8.12
19.55
25.05
PC2
RoBERTa
0.36
3.59
6.64
10.62
22.23
27.40
DeBERTa
0.85
6.00
9.52
13.32
23.06
27.91
BERT-large
-0.73
1.82
12.41
16.53
25.71
30.32
DistilBERT
-16.00
-6.33
3.19
6.54
14.80
20.86
BERT-base
-9.72
0.00
10.15
16.62
Average
RoBERTa
-9.87
-0.82
9.33
16.99
DeBERTa
-4.48
3.88
12.97
18.64
24.16
BERT-large
-5.88
3.61
12.98
18.80
24.09
Table 10: Comparison of backbones for the matching model, measured as absolute improvement in macro AP (%) over 10-shot BERT. Results are averaged across unseen classes.
Dataset
Backbone
Train time (s)
Inference time /
sample (ms)
DistilBERT
66,601
2.469
BERT
133,523
3.912
Dataset
Model
shot
Yahoo
RoBERTa
142,860
3.914
DeBERTa
241,303
6.590
Yahoo
Lbl2Transformerrer
23.68
BaRT-large
336,248
10.127
Matching
46.45
DistilBERT
5,521
2,406
Lbl2TransTormerVee
-29.14
BERT
9,832
3.724
Amazon
Amazon
RoBERTa
10,027
3.796
Matching
0.91
DeBERTa
17,089
6.073
BERT-large
26,314
9,813
PC1
Lbi2Traantormerrec
-31.29
Matching
-5.51
DistilBERT
10,613
2.474
PC1
BERT
18,812
3.757
Lb12Traanformerrec -0.29
RoBERTa
19,113
3.839
PC2
DcBERTa
33,117
6.537
Matching
11.36
BERT-large
49,581
9.883
Lbi2Trransormmrrecc -9.26
DERT
14,451
2.497
Average
BERT
25,742
3.663
Matching
13.30
PC2
RoBERTa 26,725
3.891
DeBERTa 45,435
6.405
Table 12: Zero-shot performance (macro AP %) of
BERT-large
68,481
9.851
LbI2TransformerVe and matching model on four datasets relative supervised 10-shot BERT model. We average the scores across unseen classes and three seen/unseen data splits.
Table Train inference times for different match- ing model backbones p3.1 16xlarge ec2 instance. We measured inference latency by dividing the total infer- ence time (not including time instantiate the model preprocess data) by the number of samples.

---

Dataset
Model
shot
shots
50 shots
shots
shots
1000 shots
Standard training
-7.17
0.00
7.66
11.16
15.27
16.03
MAML+ few-shot
-0.51
2.27
7.89
10.58
15.69
16.96
Yahoo
Local zero-shot
4.45
6.28
10.29
12.93
16.58
17.41
Global zero-shot
-5.67
-0.68
5.11
7.95
14.13
16.25
Standard training
-17.01
0.00
12.24
15.58
22.46
25.11
MAML+ few-shot
-00.15
-6.85
8.73
12.57
22.00
24.45
Amazon
Local zero-shot
-12.31
-1.97
10.16
12.66
21.32
24.60
Global zero-shot
29.96
-20.39
0.07
8.17
20.52
23.61
Standard training
-13.19
0,00
4.77
5.34
9.43
MAML+ few-shot
-11.88
-1.60
4,04
5.97
9.12
PCI
Local zero-shot
-9,46
0.14
4.24
5,66
9.27
Global zero-shot
-25.72
-3.92
4.36
5.73
9.04
Standard training
-1.24
0.00
3.99
8.12
19.55
25.05
MAML+ few-shot
-3.31
-2.88
2.21
6.81
16.44
21.07
PC2
Local zero-shot
4.13
-3,00
0.88
3.69
13.40
21.41
Global zero-shot
-5.65
-4.00
-2.86
-1.31
11.35
16.96
Standard training
99.65
7.16
10.05
16.68
22.06
MAML+ few-shot
-8.83
-2.26
5.72
8.98
15.81
20.83
Average
Local zero-shot
-5.36
0.36
6.40
8.74
15.14
21.14
Global zero-shot
-16.75
-7.45
1.67
5.14
13.76
18.94
Table 13: Average absolute improvement in macro AP (%) of different meta-learning strategies compared to a 10-shot matching model (BERT backbone) with standard training. We average the scores across unseen classes.
G.1 Yahoo
For the Yahoo Answers dataset the class descrip- tions were of the form "this text has to do with <class label>" where class label is one of Society & Culture, Science & Mathematics, Health, Edu- cation & Reference, Computers Internet, Sports, Business & Finance, Entertainment Music, Fam- ily & Relationships, Politics & Government". Ta- ble 14 shows examples of input texts and their labels (selected from the first 40 examples in the test set). Notice that some labels questionable (see the last two rows).
G.2 Amazon
Tables 15, 16, and 17 show example texts of prod- uets on amazon. com and their labels of the Amazon e-commerce product classification dataset. The class labels are the assigned product categories. This classification task is challenging since the dataset is web crawl data, and hence the example texts may contain misleading, irrelevant, or noisy information.

---

Example text
label
why does n optical mouse work glass table even on some surfaces optical mice use an led and camera to rapidly capture images of the surface beneath the mouse the infomation from the camera is unalyzed by a dsp digital signal processor and used detect imperfections in the underlying surface and determine motion some materials such glass mirrors or other very shiny uniform surfaces interfere with the ability of the dsp accurately analyze the surface beneath the mouse since glass is transparent and very uniform the mouse unable to pick up enough imperfections the underlying surface to determine motion mirrored surfaces are also a problem since they constantly reflect back the same image causing the dsp not to recognize motion properly when the system is unable see surface changes associated with movement the mouse will not work properly
Computers & Inter-
net
what the best off road motorcyele trail long distance trail throughout ca hear that the mojave road is amazing search for it online
Sports
what trans fat how to reduce that i heard that tras fat is bad for the body why is that where can we find it in our daily food trans fats occur in manufactured foods during the process of partial hydrogenation when hydrogen gas is bubbled through vegetable oil to increase shelf life and stabilize the original polyunsatured oil the resulting fat is similar to saturated fat which raises bad ldl cholesterol and can lead to clogged arteries and heart disease until very recently food labels were not required to list trans fats and this health risk remained hidden to consumers in early july fda regulations changed and food labels will soon begin identifying trans fat content in processed foods
Health
how many planes fodex has heard that it is the largest airlinc in the world according to the www fedex com web site air fleet 670 aireraft including 47 airbus a300 600s 17 boeing dc10 30s 62 airbus a310 200 300s 36 bocing md10 10s 2 atr 72s 5 bocing md10 30s 29 atr 42s 57 boeing mdl Is 18 bocing 727
Business & Fi- nance
100s 10 cessna 208as 94 boeing 727 200s 246 cessma 208bs 30 boeing de 10s 17 fokker [ 27s
why do people blush when they are embarrassed why do people blush when they are embarrassed from ask yahoo hitp ask yahoo com ask 200401 13 html blushing is a unique blend of evolutionary and social behavior it s an involuntary reaction of the sympathetic nervous system which is responsible for our fight or flight response hut blushing is solely triggered hy social cues people generally blush when they re feeling embarrassed scared or stressed as result of the fight or flight response the capillaries that carry blood to the skin widen and the increased blood flow lends the face as well as sometimes the chest neck or even the body or legs reddened color excessive facial blushing or erythrophobia is caused by overactivity of the sympathetic nervous system the condition can cause a lot of psychological duress and has engendered several support groups it common knowledge that animals do blush so while there are some evolutionary cues behind blushing it s also linked to something uniquely human moral consciousness
Science & Mathe- matics
is lin qingxia aka brigitte lin the most beautiful woman in chinese cinema this is according to stephen chow http www hkentreview com 2005 features kfh kfhprem html is it true who is the best lonking male star did they make any movies together well everyone has different description on what beauty is like lin qingxia hut i think many girls are prettier than she was she is more than 40 years old now if lin qingxia is the most beautiful woman in the chinese cinema the most handsome man in chinese cinema should be chin han because they always made movies together however a male movic star once was asked his girlfriend in real life or the girlfriend in movie is more beautiful he gave very good answer i think my mother is the most beautiful woman in the world
Entertainment Music
&
what s the best way to create a bootable windos dos cd i do n I use floppies any more and need to boot from something other than my hard disk well the best way is to look at whatever program you have for burning ods and see if has an option to create a bootable cd if you ca n t find it or use windows itself to burn eds then it little more complicated note if you find that booting from cd does n t work you may have to adjust your bios setting to allow your machine to hoot from cd if you want to hoot to windows the easiest way is probably to go here http www nu2 nu booted and download utility that will do it for you there are instructions there depending on what type of bont you want if you do n trust using a third party utility microsoft has some instructions here http support microsoft com kb 167685 en us this process is not very straightfrwward though if you just need to get into your filesystem and poke around you might consider booting different os for example http www freedos org freedos and hitp www knoppix net knoppix linux may do what you want for these you can download iso images and burn bootable cds
Computers & Inter-
net
what is the best riddle that you know i m trying to have a library of the best riddles that people encountered so a good riddle would be a riddle that has no ugly tricks in it pure logic answer ,AA no tricks no funky solutions if you were standing in front of a door one leading to heaven and one leading to hell neither of which are labeled and guarding that door is a guard one who always lies and one who always tells the truth but you do t know who who what question would you ask to definitively know which one is the door to heaven versus hell answer what would the other guard say that you would say is the door to heaven hell then you know the opposite door is the door to heaven hell
Business & Fi- nance
why would big ten keep its name inspite of adding a 1 11th school colleges univs xd br br upto 1980s and early 90s big ten conference had only 10 schools but then somehow it was decided to add psu as the 11th big ten school xd br br any thoughts insight on why the name of the conference not changed i would think to keep the brand recognition and the history of the big ten the conference has invested lot in building the big ten name as well as any merchandising and corporate sponsorship
Education & Refer- ence
Table 14: Examples from the Yahoo Answers dataset.

---

Example text
label
speedo essential endurance jammers aw 18mix inc polyamide and elastane machine washable endurance fabric protects against fading and retains the garments original shupe chlorine resistant ensures your swimwear wont degrade integral support hugs the contours of the body speedo essential endurance plus jammers sporty secure and comfortahle the timelessly stylish essential endurance plus jammers are perfect for fitness training constructed using endurance fabric the speedo essential jammers are hardwearing and comfortable the fabric protects against fading allowing your swimwear to look newer for longer it also clings to the body and retains its shape meaning you don't 't have to worry about baggy shorts after multiple uses a 100 chlorine resistant coating ensures your swimwear won't It degrade in the swimming pool adding to the overall durability of the product integral support is designed to hug the contours of the body to create a hyyrrodyamiic profile in the water whilst four way stretch technology allows for a wider range of motion in all directions lastly drawstring waist provides an enhanced fit and added security
Clothing. Shoes Jewelry
guide to the wildflowers of south carolina most comprehensive state plant field guide and an excellent resource for the natural history of plants in south carolina and surrounding states choice magazine richard dwight porcher is a professor of biology and director of the herbarium at the citadel in charleston south carolina an authority on the flora of south carolina he is the author of wildflowers of the carolina lowcountry and lower pee dee and co author of lowcountry the natural landscape he was born in berkeley county south carolina and received his b s from the college of charleston and ph from the university of south carolina porcher trained under dr wade batson and serves on the south carolina heritage trust udvisory board and the scientific advisory board of the south carolina nature conservancy porcher lives in mount pleasant
Books
home 2 pack value round square swirl shaped silicone mold for chocolate jelly and candy 15 piece per mold chocmolde! 301 100 food grade silicone 15 round cavities each safe to use in the oven microwave freezer and dishwasher can be used in temperatures from 76 degrees f to as high as 446 degrees now with this chocolate mold you can design your very own chocolates for gift boxes or party trays size of each cavity inch 2' c'eeatt luscious and delicious chocolate treats from the comfort of your own home now with this chocolate mold you can design your very own chocolates for gift boxes or party trays this is easy to use flexible non toxic and long lasting safe to use in the oven microwave freezer and dishwasher
Home& & Kitchen
storm trooper side white vinyl car laptop window wall decal decal automatically comes in white additional sizes available upon request made with premium
Automotive
approximate size as shown 4 2 w high quality indoor outdoor vinyl lasts for several years of interior and exterior use without fading cracking or peeling discounts available with larger quantities purchased easy to apply instructions included free tester decal included with every orderthe decal is a single color white and does not have background decals can be applied to any smooth clean surface car windows home windows interior walls laptop covers cell phone cases boats etc
navy 550lb strands cores reflective paracord parachute cord lanyard 50ft 100ftitem 550 8 cores strands reflective paracord breaking strength 550 diameter 4 mm length 50ft 100ft inner strand core tightly twisted inner strands coresthe reflective tracers appear to be light gray with normal light but in no or low light conditions the tracers will reflect back any light on them this makes the cord ideal for low light visibility conditions
Sports Outdoors
audiopipe studio 15 quot loudspeaker ohm 350w wireless stream w remotcaudiopipe dzc 1540ub studio 15 loudspeaker ohm 350w wireless stream w remoteprofessional ahs loudspeakerwoofer 15 voice coil magnet 40 oztweeter titanium diaphragm drivermax power 350 wattsvoltage 110 220vimpedance 8 ohmsensitivity 98 dbfirquencyy response 20 hz 20khzled screen usb player wireless music stream fmn radio remote controlspeaker output for passive enclosureaux for multimedia devicesblue led power indicatorwheels and handle for easy transportation
Electronics
disney goofy goof troop goofybrand new in original packagingbrand
in original packaging
Toys & Games
smartseries screen protector 3pk for the samsung r760 d710 gsii epic touchanti glare and scratch resistant washable and dust free easy to apply and remove made from durable self adheesie polymer includes cleaning cloth and an applicator cardmade from a tough self adhesive polymer the screen protector prevents fingerprints dirt dust and scratches from marking up your device the sereen protector is both easy apply and remove
Cell Phones & Ac- cessories
Table 15: Examples from the Amazon dataset, label index 0 to 7.

---

Example text
label
spy games lethal limits kindle edition
Kindle Store
seekingtag plastic heach towel clips towel holder pack of ópes (00 hrand new made durable plastic with zinc galvanized steel rust resistant also great for hanging towels to dry won't blow away or fold up includes clips in bright colors 3pes in blue 3pes in yellow size about 12 emscekingtag plastic beach towel clips towel holder pack of 6pcs
Tools Home Im- provement
geminigemini feel good with mixture of b hip hop jazz gospel for enjoy
CDs Vinyl
tarnish resistant craft wire 20 gauge gold colortarnish resistant craft wire gauge with gold color tamnish resistant spool 15 yards per spool ideal for use embellishment wirethis tamish resistant embellishment wire it is 20 gauge and gold in color there 15 yards
Arts, Sewing
Crafts
craft wire is ideal for per spool
seattle's 's best blend decaf 12 ounce pack of 2 seattle's best coffee decaf level three seattle's best blend decaf ground coffee decaffeinated ground coffee balanced smooth full flavored character profile two parts great flavor one part relavation how to enjoy measure fl 1 180ml of water add more or less coffee to achieve the perfect cup for you statements regarding dietary supplements have not been evaluated by the fia and are not intended to diagnose treat cure or prevent any disease health condition
Grocery Gourmet Food
rounded tablespoon 7g of coffee per
greenworks 15 inch 5 5 amp corded string trimmer 21272lightweight easy to use design only 7 05 lbs 15 inch cut path with pivoting head allows for edging and trimming capability with edging wheel 065 dual line auto feed with electric start gets you going within seconds no gas hassle ideal for small to medium size yards compatible greenworks replacment spool model 29082green works 5 5 amp corded 15 inch cutting width auto feed 0 065 single line electric string trimmer model no 21272
Patio, Lawn & Gar- den
2016 monthly mini wall calendar paths to god by if publishing2016 paths to god mini calendar all calendar pages are printed on fse certified paper and use environmentally safe inks size 7 in open size 7 x 14 in package quantity 1et the words of god instill peace throughout the year this paths to god mini wall calendar offers some of the most inspiring and spiritual teachings of the hible as well as images of nature's 's paths all calendar pages are printed on fse cortified paper and use environmentally safe inks
Office Products
american war generalsfor the first time national geographic gathers the nation 10 leading war generals for an unprecedented look at the history of the u s army from the vietnam war to america s war on al queda american war generals reveals many never before heard stories and opinions from the legendary leaders of the modern u army their accounts reveal the big changes that have transformed the military from the first troops to enter vietnam to the last combat troops to exit afghanistan explaining the critical personal experiences that shaped their lives and the way they approached modern warfare
Movies TV
Table 16: Examples from the Amazon dataset, label index 8 15.

---

Example
label
hooded alpaca wool dog cat knit hoodic sweater orange amp ivoryhooded striped alpaca sweater made of warm alpaca yarn perfect for dogs cats handmade by skilled artisans in peru exclusive design by alpaca warchouse made in peruthis is a brand new handmade dog and cat hooded sweater made of soft alpaca yam is very comfortable and warm and features gorgeous color combination small size measures length 14 chest girth 16 medium size measures length 15 chest girth 18 large size measures length 16 chest girth 20 xlarge size measures length 17 chest girth 22
Pet Supplies
wall mount disposable glove dispenser rackunique open design allows you to see what size or type of glove is being dispensed ideal for use in patient rooms doctor's 's offices labs kitchens morethe open design allows you to see what size or type of glove is being dispensed the holders accommodate most glove boxes and are universal enough to be used for many tissue boxes as well warning this product expose you to chemicals which is are known to the state of california cause cancer and birth defects other reproductive harm
Industrial & Scien- tific
wedgie wrpr50m 5 Omm medium wedgie rubber pick refill 18 piccesa revolutionury new sound for your acoustic or classical guitar the special elastomer material nearly eliminates pick noise leaving you with clean warm tones this pick sounds like your fingers yet plays like pick rubbers come in thicknesses and 3 levels of stiffines soft medium hard so that you can find your perfect sound mediumwedgie rubber pick refill Omm med 18pcs
Musical ments
Instru-
200 pes lot fancy colorful butterfly sticker cover for ps3 controller playstation skins games skinunit type lot 200 pieces lot package weight 0 200kg 0 44lb package size 10cm x 6em x 10cm 94in 2 36in x 3 94in unit type lot 200 pieces lot package weight 200kg 0 44lb package size 10cm x 6cm
Video Games
10em 3 94in x 2 36in x 3 94in
pack replacement maytag mfd2561 hes refrigerator water filter compatible maytag ukf8001 fridge water filter cartridgeeellaemmet maytag mfd2561hes refrigerator water filter quantity replaces ukf8001 fridge water filter cartridge refrigerator water filter retains beneficial nutrients in water while removing the taste of chlorine and odor leaving you with a fresh and clean tasting water reduces waterborne contaminants including cysts asbestos particulates lead mercury without removing beneficial minerals restricting the flow rate of the water cut down water bottle waste and save money while also helping the eavironment by using water filters for the highest quality water and best contaminant reduction replace the filter every 6 months on sale for limited time please note this is an denali pure brand replacement part not an oem product this product is not affiliated with any oem brands and is not covered under any warranties offered by the original manufacturers any warranties for this product are offered solely hy denali pure all mentions of hrand namos or model descriptions are made strictly to illustrate compatibility all brand names and logos are registered trademarks of their respective owners
Appliances
professional business card workshop Oeerette your own templates or choose layout from 000 pro- fessional quality templates premium collection of clipart images photos and heart warming sentiments to finish each project new tools for creating animations and web pages for your projects select from over 250 000 premium images and 150 creative fonts including editable clipart stunning photography and more design your own photo calendars flip books collages slide shows and photo bouquets
Software
2015 topps baseball card david wright nm mt2015 topps david wright new york mets baseball cards one single 2015 topps series trading card single card ships in top load and soft sleeve cards Art combined card condidtion near mint mint nm mt mint note stock image used2015 topps 83 david wright new york mets baseball cards
Collectibles & Fine
Table 17: Examples from the Amazon dataset, label index 16 22


Reviews:
[{"review_id": "3hFXbPF2b9", "reviewer": "Reviewer_LVKW", "summary": "", "questions": "A. How model would scale in the case of hundreds of new classes, considering for matching models, each input text needed to be calculated for each class? Any run-time analysis to reference?\n\n\n\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "I would like to start by thanking you, on behalf of all the authors, for your close reading of our paper and detailed feedback. We feel that addressing your concerns will help strengthen the paper considerably. Below we discuss your comments point-by-point.\n\n> As authors mentioned in the paper, formulating ZSL and FSL text classification as matching problem is not new to literature. Researchers have used various names for this binary classification formulation: text entailment (can be 3-class formulation as well) or similarity based approaches. In this regard, considering the proposed matching model is adopted from Yin et. al. (2019), with exception of larger token size and no pertaining strategy, the novelty from the manuscript is limited.\n\nWe agree that the model architecture is not novel. As we see it, our primary novel contributions are testing this architecture on long, complex class definitions; proposing a generalization of MAML to the matching setting to improve zero-shot performance; and comparison of the architecture against newer methods like SetFit.\n\n> Moreover, having some of recent models from similarity based approaches [SOTA, Lbl2TransformerVec, from Schopf et. al. 2022] in the experiment section would help the paper to have a stronger argument on its results.\n\nWe appreciate you bringing this paper to our attention as it is closely related to our work. We will incorporate it into the related work section of our paper. We will also test the Lbl2TransformerVec approach as an additional zero-shot benchmark, as you have suggested. This will require some modifications to our datasets, such as distilling the long class descriptions into sets of keywords. We note that to obtain reasonable averaged label keyword embeddings, we will likely have to omit potential keywords from the “not in scope” section of the class descriptions.\n\n> Authors propose MAML training to improve the ZSL performance, and demonstrate macro AP improvement over base BERT model. It would be helpful for the manuscript to compare these results with recent ZSL methods from literature as none of the baselines include any ZSL approach.\n\nAs we mentioned above, we are committed to testing the Lbl2TransformerVec approach as a zero-shot benchmark. We agree that the lack of such a model is a weakness of our current paper. Below we discuss our two motivations for not previously including ZSL benchmarks.\n\nThe first is because we seek a model which can improve with additional labeled samples. Current ZSL learning methods focus on improving zero-shot performance, and do not accommodate labeled samples from unseen classes.\n\nThe second is because we ran into difficulty creating a fair comparison between the matching approach, which is fine-tuned on seen class examples, and standard ZSL techniques, which typically are not. We tested a ZSL method based on entailment using bart-large-mnli, but found weak zero-shot performance relative to the matching approach. We expect that the performance gap is due to the fact that the matching approach is fine-tuned on domain-specific data in the form of seen class examples, whereas BART is not. Therefore the comparison is not entirely fair. Perhaps a more fair comparison would be to fine-tune BART on the seen class examples before evaluating on unseen ones. However, this is effectively the matching approach we tested, but with a larger base transformer model. Since we already showed that larger models improve matching model performance further, we felt it was unnecessary to perform this additional experiment.\n\nWith that being said, we will test and compare Lbl2TransformerVec with the matching approach in a zero-shot setting.\n\n> Moreover, although it is reasonable, the choice of macro AP is unusual for ZSL and FSL tasks. It would also add more insight to their experiment section to show the comparisons using averaged class accuracies, for both seen and unseen classes. This way, class imbalance would not be an issue. Having both seen and unseen classes averaged accuracies on tables would also help to better understand the models overall performance. The recent format (except Table 7 from appendix) does not convey the whole message. When dealing with ZSL or FSL, it is not unusual for models to favor one side and have limited ability on other side (unseen vs seen classes).\n\nWe considered accuracy, but found that it gave misleading results due to the large number of classes. For example, the PC1 dataset has 50 classes, most with positive rates below 3%. As a result, a model predicting 0 for every sample is able to achieve about 97% class-averaged accuracy. We found that macro AP gave a more informative measure of model performance.\n\nWe agree that having model metrics on the seen classes would be ideal, but we faced two practical constraints, which caused us to omit them. First, we worried that putting additional entries in the tables would make them too dense and take up too much space. Second, computing seen class performance would significantly increase the cost of the supervised baseline models (especially BERT, BERT+Adapter, SetFit). We were able to reduce the compute cost of training and evaluating these models by 80% by applying them only to the unseen classes (20% of the overall classes).\n\n> One of the main contributions of the paper is about adjusting the proposed model to better handle the long class descriptions. One might challenge the need for such set-up, considering the complex product descriptions (long class descriptions) can be simplified into small, brief and concise text fragments using LLMs (GPTs).\n\nThis is a valid criticism. In some cases it is possible to summarize or otherwise shorten complex class descriptions, whether with an LLM or manually. However, there will remain cases in which descriptions cannot be further compressed. For example, the class descriptions for our PC1 and PC2 datasets were written by subject matter experts, and are unlikely to be easily shortened without losing valuable information. Therefore we believe that our work is still useful.\n\n> Question: How model would scale in the case of hundreds of new classes, considering for matching models, each input text needed to be calculated for each class? Any run-time analysis to reference?\n\nThe model scales linearly with the number of classes, so doubling the number of candidate classes, for example, will double the inference cost. With N classes and M examples, we must compute N*M embeddings.\n\nThis is one of the primary reasons we are currently testing bi-encoders. With a bi-encoder, each example text and class text need only be embedded once, for a total of N+M embeddings. In the worst case we still must compute N*M similarity calculations, but these are much cheaper than transformer-based embeddings.\n\nWe would be happy to add a discussion of these considerations, if you think it would strengthen the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is easy to read and follow.\n\nAuthors show that casting the ZSL and FSL text classification as matching problem is an effective solution even if the class descriptions are quite long. \n\nThey also devised a strong baseline using MLPC by generalizing prototypical networks.\n\nThe experimental results on Macro AP validate their proposed matching algorithm's better performance compared to baselines.\n\nThe proposed model can have a smooth shift from low data regime to high data regime.", "reasons_to_reject": "As authors mentioned in the paper, formulating ZSL and FSL text classification as matching problem is not new to literature. Researchers have used various names for this binary classification formulation: text entailment (can be 3-class formulation as well) or similarity based approaches. In this regard, considering the proposed matching model is adopted from Yin et. al. (2019), with exception of larger token size and no pertaining strategy, the novelty from the manuscript is limited. Moreover, having some of recent models from similarity based approaches [SOTA, Lbl2TransformerVec,  from Schopf et. al. 2022] in the experiment section would help the paper to have a stronger argument on its results.\n\nAuthors propose MAML training to improve the ZSL performance, and demonstrate macro AP improvement over base BERT model. It would be helpful for the manuscript to compare these results with recent ZSL methods from literature as none of the baselines include any ZSL approach. Moreover,  although it is reasonable, the choice of macro AP is unusual for ZSL and FSL tasks. It would also add more insight to their experiment section to show the comparisons using averaged class accuracies, for both seen and unseen classes. This way, class imbalance would not be an issue. Having both seen  and unseen classes averaged accuracies on tables would also help to better understand the models overall performance. The recent format (except Table 7 from appendix) does not convey the whole message. When dealing with ZSL or FSL, it is not unusual for models to favor one side and have limited ability on other side (unseen vs seen classes).\n\nOne of the main contributions of the paper is about adjusting the proposed model to better handle the long class descriptions. One might challenge the need for such set-up, considering the complex product descriptions (long class descriptions) can be simplified into small, brief and concise text fragments using LLMs (GPTs). \n\n\n\n\n\n\n\n\n\nSchopf et al 2022 . Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches", "questions_to_authors": "A. How model would scale in the case of hundreds of new classes, considering for matching models, each input text needed to be calculated for each class? Any run-time analysis to reference?\n\n\n\n", "paper_topic_and_main_contributions": "The paper is about providing a unified solution for zero and few shot learning problems in text classification by reformulating the assignment as a matching problem.\n\nThe main contributions are three folds:\n\n(1) Experimentally validating that matching formulation exhibits strong results when class labels have long descriptions\n(2) Extending the Model-Agnostic Meta Learning to improve the ZSL performance.\n(3) Matching formulation allows to extend the number of new classes as needed given class descriptions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Schopf et al 2022 . Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches", "Typos_Grammar_Style_and_Presentation_Improvements": "The paper is well written."}, "extra_text": {}}, {"review_id": "awTLw1sIyS", "reviewer": "Reviewer_LCiA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 1, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> While the paper conducts two experiments to test the models in zero- and few-shot scenarios, the paper could benefit from additional experiments to further validate the proposed approach. For example, the paper only focuses on methods that use semantic attributes, so it might be interesting to explore other zero-shot techniques that rely on different approaches, such as inter-class correlations or joint embeddings of classes and examples.\n\nWe sincerely appreciate your thorough review of our paper. Your constructive comments have provided us with valuable insights into areas where our work can be further improved. In particular, we are grateful for your suggestion to consider additional experiments and explore alternative zero-shot techniques. Regarding techniques benefitting from additional information such as inter-class correlation, we are familiar with Rios and Kavuluru (2018), which utilizes class/label hierarchies with a DAG, as well as Zhang et al. (2018); Yang et al. (2018); Xun et al. (2020), which delve into learning inter-class correlations from labeled examples. However, it is crucial to note that our specific experimental setup and dataset do not possess a natural hierarchy of classes/labels. Moreover, our scenario, which focuses on new and unseen classes with limited or no labeled examples, limits the applicability of methods that rely on abundant labeled data to learn inter-class correlations.\n\nNevertheless, we agree that exploring joint embeddings between classes and examples holds significant promise for advancing zero-/few-shot text classification performance. We have explored related work such as Pappas and Henderson (2019), which learns joint input-label embeddings. And it motivated us to develop our proposed matching model, which is more general, less constrained (without a non-learnable joint bilinear input-label representation), and with a larger encoder (transformer instead of MLP and GRU). We are committed to investigating this direction further and incorporating Lbl2TransformerVec from Schopf et. al (2022), which builds a joint embedding space of examples and classes/labels.\n\nSchopf et. al (2022) - Evaluating Unsupervised Text Classification: Zero-shot and Similarity-based Approaches\n\n> The proposed model in the paper uses long text (class description) to measure similarity, but it has a limitation of text length being limited to 512 tokens due to the underlying language model. This limitation may affect the model's performance on longer texts.\n\nWe admit that the proposed model has a limitation of text length (cannot exceed 512 tokens). This could be easily resolved by changing the backbone to a model that takes longer input sequence but we could not test with other architectures with longer input sequences. We will add this in the limitation section of the paper.\n\n> The study only used modest-sized language models with 110 to 184 million parameters, so the results may not be applicable to much larger or smaller models.\n\nThank you for pointing this out. We are currently experimenting with models of different sizes: DistilBERT (66M params, 60% of BERT-base) and BERT-large (330M, 300% of BERT-base). We will update the result in appendix section in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper's approach to text classification include its ability to handle new classes from the beginning (zero-shot, cold-start) to the end of the model development cycle (few- to many-shot) by exploiting long, complex class descriptions and leveraging both class descriptions and standard labels. The proposed matching models achieved 13.30% macro AP with zero-shots (relative to 10-shot BERT) and surpassed few-shot baselines by 22.48% with 10-shots. The extended Model-Agnostic Meta-Learning algorithm also improved zero-shot performance by a further 4.29%. In conclusion:\n\n1. The proposed approach to text classification that leverages labels and complex class descriptions to perform zero- and few-shot learning on new classes.\n2. Strong zero-shot performance and scalability with few-shot samples, beating strong baselines by 22.48% in the 10-shot setting.\n3.. Comparison with numerous baselines on text classification tasks with complex class descriptions.\n4. The potential for advancing the state-of-the-art in text classification and the practical applications of the proposed approach in various NLP tasks.\n\n\n", "reasons_to_reject": "While the paper conducts two experiments to test the models in zero- and few-shot scenarios, the paper could benefit from additional experiments to further validate the proposed approach. For example, the paper only focuses on methods that use semantic attributes, so it might be interesting to explore other zero-shot techniques that rely on different approaches, such as inter-class correlations or joint embeddings of classes and examples. \n\nThe proposed model in the paper uses long text (class description) to measure similarity, but it has a limitation of text length being limited to 512 tokens due to the underlying language model. This limitation may affect the model's performance on longer texts. The study only used modest-sized language models with 110 to 184 million parameters, so the results may not be applicable to much larger or smaller models. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This article presents a new approach to text classification that leverages labels and complex class descriptions to perform zero- and few-shot learning on new classes. The proposed approach overcomes the shortcomings of previous work in text classification and shows strong zero-shot performance and scalability with few-shot samples, beating strong baselines by 22.48% in the 10-shot setting. The article also discusses the limitations of the proposed model and the ethical considerations taken into account during the research.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "RvCuPM1bcS", "reviewer": "Reviewer_6T3q", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to express our gratitude and thank you for your valuable comments and suggestions. Your insightful feedback has greatly contributed to our understanding of paper’s strengths and potential areas for improvement. We are also pleased to see that you acknowledge the significance and extensiveness of our analysis and experiments. In the following, we address the weak points and reasons to reject that you raised.\n\n> The main weakness of the paper, and perhaps the only one, is that the methods considered by the paper are well-known.\n\nWe admit that the methods we consider e.g. matching, prototypical networks, and sentence transformers, are well-known SOTA approaches for various zero-shot or few-shot classification problems. However, we were motivated to conduct extensive experiments as it was unclear from literature which method with what adaptation can give us (a) best overall performance across zero-shot, few-shot, and many-shot regimes, and (b) scalability to many new unseen classes. We believe that a comprehensive analysis of SOTA methods with adaptations is necessary, and our paper sheds light on further advancements in zero-/few-shot text classification by leveraging long, complex class definitions from both seen and unseen classes as well as labels from seen classes.\n\n> Matching is a well-known strategy for classification and it has been used before, and the only proposed change to the model is to fine-tune on the few-shot dataset, which is also a standard practice for few-shot classification.\n\nWhile fine-tuning on few-shot dataset is indeed a common practice, it has certain limitations such as overfitting and catastrophic forgetting (unable to scale to many new classes) as described in our paper. We explore the matching setting to increase its generalizability from seen classes to unseen classes. We also propose a novel extension of MAML to zero-shot learning, which is applicable to any matching model.\n\n> The adaptation of MAML is itself novel but it seems to be mainly a trick that's made possible by the specific architecture adopted, and not a new paradigm.\n\nThank you for pointing this out. While our enhancement to MAML is somewhat limited in scope, it is actually applicable to a wider variety of models than may be first apparent. It is compatible with any matching model architecture with which MAML also works. The only constraint that our approach adds, beyond those already imposed by MAML, is that the model should have a fixed number of classification outputs. That is to say, predictions for unseen classes should be extracted from the fixed number of outputs somehow, e.g. by formulating classification as a binary matching problem. This constraint still permits one to use any neural network. In our revised version, we will place greater emphasis on the specific context and generalizability of MAML adaptation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Not only the problem is clearly timely and important, but the paper has extensive and thorough experimentation, comparing several baselines, some of which they improve upon as a novel contribution. They compare both different datasets, with different levels of difficulty, different few-shot regimes, and different training procedures. The choice of metric also reflects care of understanding the problem space and how models will be used in practice.\nHence, it is quite likely that the results are meaningful for researchers and practitioners in this space.", "reasons_to_reject": "The main weakness of the paper, and perhaps the only one, is that the methods considered by the paper are well-known. Matching is a well-known strategy for classification and it has been used before, and the only proposed change to the model is to fine-tune on the few-shot dataset, which is also a standard practice for few-shot classification. The adaptation of MAML is itself novel but it seems to be mainly a trick that's made possible by the specific architecture adopted, and not a new paradigm.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper considers the problem of zero- and few-shot text classification, and whether human provided detailed class descriptions, which are typically used in zero-shot classification, can also be applied to few-shot classification. Through extensive experimentation in different few-shot settings and across different datasets - two public and two internal - the paper shows that a matching model, consisting of a finetuned PLM that encodes both the input text and the class description and computes a matching score, achieves the best overall performance. The paper also extends meta-learning training to the zero-shot case, by introducing a novel sampling strategy for unseen query classes.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper reframes the text classification tasks, particularly in the zero-shot and few-shot settings, as matching class descriptions. They show strong performance on a number of tasks using many baselines. They also extend the Model-Agnostoc Meta Learning algorithm (MAML) in this context and show improved performance.

The reviewers noted several limitations with respect to evaluations only on tasks of limited token length (so affects length of class descriptions), focusing on tasks with semantic classes, and also using language models with fewer parameters. The authors have addressed some of these points in the rebuttal. Getting final soundness scores of (4,3,5) and excitement scores of (4,3,3)
Decision: Accept-Main