Anchoring Fine-tuning of Sentence Transformer with Semantic Label Information for Efficient Truly Few-shot Classification
Authors: Amalie Brogaard Pauli
Leon Derczynski
Ira Assent
Venue: EMNLP 2023 Main (2023)
Keywords: few-shot
sentence transformer
classification
efficiency

Abstract: Few-shot classification is a powerful technique,
but training requires substantial computing
power and data. We propose an efficient
method with small model sizes and less training
data with only 2-8 training instances per class.
Our proposed method, AncSetFit, targets low data
scenarios by anchoring the task and label
information through sentence embeddings in
fine-tuning a Sentence Transformer model. It
uses contrastive learning and a triplet loss to enforce
training instances of a class to be closest
to its own textual semantic label information
in the embedding space - and thereby learning
to embed different class instances more distinct.
AncSetFit obtains strong performance
in data-sparse scenarios compared to existing
methods across SST-5, Emotion detection, and
AG News data, even with just two examples
per class.

Anchoring Fine-tuning of Sentence Transformer with Semantic Label Information for Efficient Truly Few-shot Classification
Amalie Brogaard Pauli Leon Derczynski Ira Assent
*Department of Computer Science, Aarhus University, Denmark 2IT University of Copenhagen, Denmark (ampa, jecs. ausss.aau.r au. dk, ld@itu. dk
Abstract
data. To do this, we propose adding semantic infor- mation about the task and labels using fine-tuning of sentence transformers (Reimers and Gurevych, 2019). Through this, we inform the model of what is to be classified by adding textual information. We use contrastive learning and triplet loss to en- force instances from different classes to be closest to their own class' textual description. Additionally, (RQ2) we ask to what extent variation in textual information affects model performance. Our con- tributions include:
Few-shot classification is powerful technique, but training requires substantial computing power and data. We propose an efficient method with small model sizes and less training data with only 2-8 training instances per class. Our proposed method, AncSetFit, targets low- data scenarios by anchoring the task and label information through sentence embeddings in fine-tuning Sentence Transformer model. uses contrastive learning and triplet loss to en- force training instances of a class to be closest its own textual semantic label information in the embedding space and thereby learning embed different class instances more dis- tinct. AncSetFit obtains strong performance in data-sparse scenarios compared to existing methods across SST-5, Emotion detection, and AG News data, even with just two examples per class.
AncSetFit, method that adds semantic infor- mation about the task SetFit while main- taining the same efficiency.
Empirical evidence that AncSetFit gives higher performance in extremely few-shot classification settings.
1 Introduction
Code is publicly available at https://gi thub. com/Amal htw.mmmiieaalllAaStafeaiil.
Data sparsity remains problem for training NLP models. Pre-trained language models (PLMs) have been adapted to NLP downstream tasks using only a few ground truth examples. However, many of these methods require substantial computing, re- ducing their practicality. The SetFit method (Tun- stall et al.. 2022) offers promising route to few- shot text classification, due to its computational efficiency and relatively low data requirement. The method is prompt-free, which is an advantage in that it skips the variation inherent to prompt-based learning methods since these depend on choosing good/multiple prompts. On the other hand, in a truly few-shot setting, providing the model with information about the labels it should generalise to from the few given training examples is expected be helpful (Schick and Schütze, 2020).
2 Background
2.1 Related work
Few-shot text classification reduces the need for annotated data, which is often costly and time- consuming to obtain. It has gained increased at- tention with large pre-trained language models (PLMs), especially GPT-3 with in-context learn- ing, which has no parameter update to the model, but a model size of 175 billion parameters (Brown et al., 2020). Prompting is also used in Pattern Exploiting Training (PET) (Schick and Schütze, 2020), which uses the training objective of the lan- guage model with cloze-like questions to predict task. ADAPET extends PET for improved per- formance and eliminating the need for unlabeled task-specific data (Tam et al., 2021), based AL- BERT (Lan al.). PERFECT is likewise a PET- based method, which improves efficiency using task-specific adapters and eliminates the need for
We ask (RQ1) if the efficiency small models in SetFit can be retained while also making the method label-aware and this improves perfor-
mance, especially in cases with really litle training

---

handcrafted verbalizers and patterns (Mahabadi al., 2022).
We see the benefit of adding task description in- formation, as seen in PET methods. We, there- fore. propose to inject information about the task into the method, such that it can help separate the embeddings the desired directions for a task. Intuitively, we want to provide the model with good class-specific starting points in the embed- ding space for sentences to cluster around. We think of them as starting points since we want the class-specific points be learnable as well.
Recently, there has been work to reduce runtime and model sizes. The ootte-of-the-att in few-shot classification is T-Few (Liu et al., 2022) which is parameter-efficien fine-tuning (PEFT) method based on TO model (Sanh et al., 2021). Although T-Few much smaller than GPT-3, it is still large with billions of parameters. A method that is truly efficient in terms of computing requirements the SetFit method, which uses a backbone model only in the sizes of millions of parameters (Tunstall et al.. 2022). In our work, we build on SetFit to address challenging few-shot classification tasks efficienlly.
We introduce anchor statements, which are sen- tences containing semantic information for each class i.e. a task/class description. As they are textual, they can be embedded with the model and hence updated under fine-tuning. We look to the Sentence Transformer paper (Reimers and Gurevych, 2019), which experiments with triplet loss function for sentence similarity in articles. The input is triplet of three sentences, two from the same article, and one from a distinct, and then the loss minimises the distance between the same class pair. We propose to modify the input of the loss function to be class-specific fixed anchor statement, and two samples from distinct classes, one matching the class of the anchor statement thus getting the semantic label information into the loss function. The objective is ensure that the samples of one class are closer their own fixed class-specific anchor statement, hence getting the embeddings to separate in the desired direction.
2.2 SetFit: An efficient method
SetFit (Tunstall al., 2022) uses pre-trained Sen- tence Transformer (Reimers and Gurevych, 2019). The sentence transformer embeds entire sentences/- paragraphs one embedding space based on simi- larity measures and siamese and triplet networks. SetFit has two phases: 1) it applies contrastive learning to fine-tune a sentence transformer model. It augments training data by generating triplets from the few available training samples: drawing pairs of samples from the same class and pairs from distinct classes and minimizing the loss func- tion || [u, v) - d()(w), e(u))llz where d(...) the cosine distance function, e(-) is the embedding, v are input samples and the indicator function I (u,, = (1 if L(u)= = L(v) else O, where L is a mapping function to a class; so indicates the two samples are of the same class (positive) and 0 the samples are of different classes (negative). Intuitively, this clusters embeddings such that em- beddings of different classes should be far apart, and those of the same class close in the embedding space. 2) SetFit uses a traditional classifier on the embedding vectors.
More formally, we generate input triplets (a, u, v) by drawing samples u, v from two distinct classes and adding the anchor statement of one of the classes. We use the loss function:
d(e(a"), e(v")) 0,
where is indicating samples of the same class (positive), and of another class (negative), d(.,)) :) is the cosine distance function, e(-) is the embed- ding. and the margin is hyperparameter. Essen- tially, we want to achieve d(e(a"), (eeaa"),((u))) + d(c(a"), e(v")). meaning that the sentences from a class should be closer to the anchor statement i.e the class description of that class than a sentence from different class should be. Therefore, the is the margin to the anchor point we want to enforce between samples of diffferent classes. Note that dur- ing training, the embedding of the anchor statement is also updated, and the pre-set "centre points' can therefore adjust accordingly. For intuition, see also a visualisation on fine-tuning two datasets using PCA in Appendix A.
3 AncSetFit: Anchoring SetFit with Semantic Label Information
The intuition in SetFit, as discussed in the sec- tion above, is to get the embeddings to separate class-wise before inputting them into classifier. However, if the training data is really small, there might be lack of information about the classes that would allow the embedding to cluster accordingly and thus to generalise. When only a few samples are available, the samples might have other prop- erties in common than what the task prescribes.

---

Emotion
Yelp
80
60
90
50
60
Accuracy
50
Accuracy
84
45
50
45
40
35
30
35
66
24.8
32
64
24.8
16
32
64
248
16
32
64
24.
32
64
Questions
SemEvai2018Taskt
90
84
78
Accuracy
60
66
63
72
66
48
60
SetFit
AncSetFit
24.8
16
32
64
26.
16
32
64
4116
32
64
128
Figure I: Average accuracy for the single-label tasks and Fl-micro for the multi-label task, and + std deviation for different training set sizes for comparing SetFit and AneSetFit. Training data sizes are per class.
4 Experiments
News (Greene and Cunningham, 2006). see Sec- tion 6. After the development stage, we have one go at the test datasets.
4.1 Data
We conduct experiments text classification data, where several have been used in mimic few-shot settings (Min al., 2022; Gao al., 2021; Tun- stall et al., 2022). true few-shot setting, data for developing and tuning hyperparameters would not be available. Thus, as in Tunstall et al. (2022), we split the data sets into development and testing. All data are in English, mainly in sentiment anal- ysis and topic classification; mostly single-label classification, plus few multi-label tasks.
We simulate the few-shot setting by drawing samples with balanced labels, as in Gao et al. (2021); )); Tunstall et al. (2022) randomly from the original training data, repeated 10 20 times for each experiment, account for possible variance in data quality. Finally, we test our method on the real-world few-shot classification benchmark RAFT (Alex et al., 2021) 2 with 11 tasks, each 50 training samples.
4.2 Testing Setup and baselines
Customer Review (CR) (Hu and Liu, 2004); Stanford Sentiment Treebank- (SST5) (Socher et al., 2013); Emotion (Sar- avia et al., 2018); AG News, Yahoo An- swer Topics (Yahoo Questions) and Yelp Re- view (Zhang et al., 2015); multi-label Se- mEyal20028Taat (Mohammad et al., 2018).
Data for testing:
Mainly, we benchmark against the SetFit method since AncSetFit is proposed alternative, and study the effect of our proposed anchor statements in isolation. Note that both methods use the same relatively small backbone model and are equal in terms of efficiency. Appendix D gives training details and hyperparameters, Appendix C anchor statements.
For further details on data and tasks see Ap- pendix B. In total, we use seven different datasets for development and hyperparameter tuning but stay as similar to SetFit as possible for compar- ison. We experiment with different hyperparam- eter settings, such the margin parameter and anchor statements on development data. We re- port experiments examining the effect of different anchor statements on the following development dataset SST2 (Socher et al., 2013), IMDB (Maas et al., 201 Student Question Categories, BBC
Method backbone
Size
PERFECT roberta-large ADAPET XXL a-bert-xxlarge-v2
355M
223M
ADAPET BASE albert-base-v2
117M
SetFit paraphraas--mpne--aae-y AncSetfit paraphrase
110M
110M
Table 1: Backbone model size in terms of number of parameters. Backbone models are downloaded from HuggingFace, URLs in Appendix D
wtg.gp/..w.o...cccoomommmoaaaaae suudents-questions-dae

---

Size
PERFECT
ADAPET BASE
ADAPET XXLARGE
SetFit
AncSetFit
Emotion
22.1 16.0
34.5 16.7
46.7 17.0
30.0 16.2
49.1 14.7
4
22.5 13.5
45.0 65.6
48.6 14,9
39.1 14.2
53,2 #3.2
30.2 +6.8
50.9 +3.6
53.9 +3.9
49.9 +4.0
56.6+2.6
27.3 +3.1
+3.2
47.0 #3.2
33.4 +2.8
40.5 +3.0
SSTS
30.3 +5.5
35.5 1200
48.1 #32
38.4 12.0
43.4 #2.1
34.8 +2.7
+2.0
49.7 #2.0
42.8 +3.4
45.2 +2.0
63.7 +8.6
76.8 +10.1
87.2 +5.3
75.5 +12.6
86.6 +3.9
CR
75.6 +7.7
77.0 18.3
90.6 =2.5
86.8 14.2
89.8 11.8
82.7 19.2
78.616.1
90.6 1.7
89.6 11.5
90.7 11.4
AG News
2
51.8 -22.
64.5 12.6
76.8 16.5
65.3 16.2
76.0 14.2
67.2 +8.3
73.1 #3.2
84.1 =2.0
76.5 #3.6
81.6 #1.9
79.2 15.6
77,9 13.3
86.0 128
82.8 12.4
84.2 11.
Yahoo Q"
2
28.1 +4.9
48.8 +3.1
51.1 #3.2
42.9 +5.2
57.1 +2.3
40.3 +4.7
56.6 +2.1
59.0 #1.8
55.4 +2.7
62.3 #1.4
54.2 +2.9
59.0 +1.8
63.5 11.
61.0 +1.9
65.3 +1.4
Yelp R
24.7 +3.8
32.0 +446
50.9 +1.2
36.4 +4.3
48.1 +2.3
30.5 +5.0
40.8 13.3
54.7 12.
44.9 14.8
51.6 11.4
39.1 +3.5
44.0+3.1
58.2 116
50.7 +3.4
53.9 +1.5
Table Average accuracy, standard deviation for 2,4 and 8 training samples per class. YYahoo Q: Yahoo Questions,
Yelp R: Yelp Review
AncSetFit is targeting small model sizes and ex- treme few-shot settings, we therefore also provide benchmark results on ADAPET (Tam et al., 2021) and PERFECT (Mahabadi et al., 2022) when train- ing on 2-8 instances per class. ADAPET and PER- FECT are chosen as benchmarks as these methods also use relatively small back-bone model sizes in terms of the number of parameters, Table 1. ADAPET originally used ALBERT xxlarge-v2 as the backbone, but in order compare the meth- ods with similar model sizes, we also benchmark again ADAPET using ALBERT base-v2. With ADAPET we maintain the same level of informa- tion provided to the model as in AncSetFit, with the information from the anchor statements cor- responding to the verbalisers and patterns. Note PERFECT and ADAPET originally use more data, PERFECT is developed using 32 training instances per class. ADAPET originally assumed access to development set, which is disabled here, similar to the comparison done Tunstall et al. (2022). The experiments are repeated 20 times for AncSetFit and SetFit, 10 times for ADAPET and PERFECT, Implementation and training details are provided in Appendix D.
Results and Discussion
We compare AneSetFit to SetFit on seven diffcrent datasets in Fig. 1, with selected details Table 2. On all seven datasets, we see that AncSetfit outper- forms SetFit in terms of accuracy when there are only 2-8 samples per class available for training, thus demonstrating its benefit for the most extreme cases. On larger sample sizes, the gain decreases. This result consistent with the Ablation study of the anchor statements in Section 6: fine-tuning the sentence transformer benefits from anchoring information when the training data size is small, and for larger training sizes the method becomes less and last no longer dependent on it. We no- tice large numeric differences in accuracy in the extremely few-shot cases; e.g., after training on only two samples times six classes in the Emotion dataset, AneSetFit achieves an average accuracy of 49.1 against the lower 30.0 for SetFit. We also observe lower standard deviation across datasets for AncSetfit than for SetFit, which means that it is less dependent getting good training samples. As AncSetFit only holds an advantage per- formances over SetFit when training data sizes are low, we compare AncSetFit to ADAPET and PER-

---

FECT in the training size range of 2-8 instances per class. (Comparing SciFit to ADAPET and PER- FECT for larger training sizes we refer the reader to (Tunstall et al., 2022).) AncSetFit outperforms PERFECT and ADAPET_ DAAAETTASEEE.EE on all six datasets for training sizes 2,4,8 in Table 2. Comparing An- cSetfit to ADAPET_NN XXL, we get more varying results. AncSetFit is superior the Emotion de- tection task and the topic classification in Yahoo Questions, whereas ADAPET _XL is on SST5, AG News and Yelp Review.
IMDB
SST2
90
80
70
60-
60
50
40
4
26
32
16
32
BBC News
Student Question Cat.
90
80-
60
70
However, ADAPET_XAB XXL model size is double that of AncSetfit in terms of parameters Table I, and AncSetFit completes both training and infer- ence in seconds, whereas ADAPET_XOL XXL takes hours in our setting. Numbers listed in Appendix Thus, AneSetFit both efficient and effective.
50
w.Labels
w:Synonyms
Figure 2: Ablation study: average accuracy, + std devi- ation for different training sizes and anchor statements. "wisynonyms' lines omitted to avoid overplotting.
RAFT leaderboard AnSetFit achieves higher performance in 4 out of 11 RAFT tasks than Set- Fit. SetFit shows a higher average accuracy. The "Banking_ 77' dataset has classes with training samples, which makes unsuitable for supervised learning with AncSetFit. Further, the majority of the datasets are binary with more labels per class than in the range where our experiments show an acceleration in results. Lastly, we note that several tasks are semantically complex, in that the descrip- tion of the classes is rather long, making it more difficult to capture the information in the anchor statement. For example, the task 'systematic review inclusion' has specification list of criteria for the classes. However, AncSetFit gets higher performance on the two Twitter tasks of detecting hate speech and complaints.
is better than providing it with no information (let- ters), 2) confusing the model with opposite labels is worse than no information. Both indicate that the model indeed benefits from semantic informa- tion provided in the anchor statements. Especially, observe large numeric differences in the accu- racy score for training sizes of 4 and 8. Thus, in the more extreme few-shot settings, the anchor statements in AncSetFit provide the greatest ben- efit. The larger the training set size, the less the model depends on correct semantic information in the anchor statement, as the performance gaps close. Intuitively this makes sense, as the more training data the easier to infer the commonalities of the classes. We also observe, in the two tasks of sentiment analysis, that providing synonym words does not have a large effect, indicating that the method robust to small semantic changes in the template.
6 Ablation Study: Effect of the Anchor Statements
In the development phase, we study the impact of the anchor statements. We construct the anchor statement in the generic form where a template is specific to each task and a word describing the class. We evaluate four ways of constructing w; 1) words of the labels, 2) single letter representing no actual information, 3) permutated label words from such that the label word does not match the class, and 4) label synonyms (see Appendix B Table 4 for templates). Figure 2 shows the results for 4.8.1 and 64 training samples per class, averaged over runs. In all four tasks we ob- serve for small training set sizes: 1) providing the model with the correct semantic label information
Conclusion
This paper proposes AneSetFit, an efficient method for extreme few-shot text classifications on small model sizes. The method injects semantic informa- tion about the task and labels to fine-tune Sen- tence Transformer. The method separates samples of different classes in the embedding space by pro- viding anchor points of textual information about the class labels cluster around. We show in number of tasks for sentiment and topic classifica- tions that AncSetFit achieves higher performance in the extreme few-shot settings where there are fewer than eight training samples per class.

---

Limitations
and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers). pages 38 6-3380.
The method is limited to text classification tasks, where the task and corresponding classes can some- how be described with short opposing textual state- ments. Therefore, when tasks become more se- mantically complex, i.e., not easily described in a sentence, then the method approaches its lim- its. Additionally, the method's advantage is the relatively high performance using small backbonn model sizes, but of course, like SetFit it still has performance gap to large-scale models (Tunstall al., 2022).
Derek Greene and Pádraig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. Proceedings of the 23rd international conference Machine learning. pages 377-384.
Minqing Hu and Bing Liu. 2004. Mining and summa- rizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pages 168-177.
Zhenzhong Lan, Mingda Chen, Sehastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self sulf-supe learning of lan- guage representations. In International Conference on Learning Representations.
Ethics Statement
As ever, generalisations drawn from fewer points of data have higher risk of being inaccurate. While methods like AncSetFit that improve quantitative performance on benchmarks based on fewer data points represent advances in data efficiency, it is important not to over-interpret higher scores as an indication of excellent broad-domain generalisa- tion.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin A Raf- fel. 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Ad- vances in Neural Information Processing Systems. 35:1950-1966 1965.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arxiv: 1907.11692.
Acknowledgements
This work was supported by the Danish Data Science Academy, which is funded by the Novo Nordisk Foundation LUM FONDEN (40516).
and VIL-
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
References
Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, et al. 2021. Raft: A real-world few-shot text classification benchmark. arXiv preprint 1rX9::2109.144
Rabech Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Sacidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt- free and efficient few-shot learning with language models. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3638-3652.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022. Noisy channel language model prompting for few-shot text classification. Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 53 5316-5330.
Dorottya Demszky, Dana Movshovitz-Attie Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020. GoEmotions: A Dataset of Fine-Grainee Emo- tions. In 58th Annual Meeting of the Association for Computational Linguistics (ACL).
Saif M. Mohammad, Felipe Bravo Marquuz Mo- hammad Salameh, and Svetlana Kiritchenko. 2018. Semeval-2018 Task I: Affect in tweets. In Proceed- ings of International Workshop on Semantic Evalua- tion 2002188)20018) New Orleans, LA, USA.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics

---

Nils Reimers and Iryna Gurevych. 2019. Sentence-bers Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing TEMNLP-IJCNLPP pages 3982-3992.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training ables zero-shot task generalization. arXiv preprint arXiv:21 10.08207.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. CARER: Con- textualized affect representations for emotion recog- nition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2020. Exploit- ing cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning. Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference empiri- cal methods natural language processing, pages 1641-
Derek Tam, Rakesh R Menon, Mohit Bansal. Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4980 4991.
Lewis Tunstall, Nils Reimers, Unso Eun Seo Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts. arXiv preprint arXiv:2209.1 1055..
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019. Huggingface's transformers: State-of- the-art natural language processing, arXiv preprint arXiv:1910003771.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances in neural information processing systems, 28.

---

A Appendix: Visualisation of the method
We illustrate the AncSetFit and the effect of its anchoring approach through PCA on the Emotion and CR datasets. In each task, we illustrate the two principal components of the embeddings 1000 test samples before fine-tuning the Sentence Transformer model, and after fine-tuning with the AnchSetFit approach on only instances per class. We show the anchor statements in the plot both before and after fine-tuning, On both the Emotion and the CR datasets, can observe that the samples and anchor statements of the different classes separate more the space after fine-tuning. Visually, we can imagine how the anchor statement is pushed apart and how the samples are drawn towards the anchor statement. Illustrating how we expect the method to work.
(a) Emotion, before fine-tuning
(b) Emotion, after fine-tuning on 4 instances per class
83
(c) CR, hefore fine-tuning
(d) CR, after finc-tuning on 4 instances per class
Figure 3: Illustration using PCA embeddings of the AneSetFit mehtod
B Appendix: Datasets
Table lists the datasets used respectively in the development phase and for testing. We provide a short task overview, the number of classes and the support the testing set, the *) means we

---

have limited the test-set size. The majority of datasets are downloaded from https://huggingfaac co/SetFit, where we follow any split or partition of the data provided here. Other sourees of data are 1) Yahoo Questions downloaded https://huggiingacce co//ataseet///yhooooassssrr
topics, 2) Abstract which is from a Kaggle competition at https: /kaggle com//datasetssvinn233. mmntthackk-indepeddencce sandeepeddenceeddy-200808 111200m1-hccktthor and here the test set is taken out of the train part for the experiments. 3) the english part SemEval2018 8Task] is origional from Mohammad et al. 1. Notice,
(2018) and downloaded from https: //tp::///hgggnnggg co/aatasts// eeaaleellllllleeeee 8_task_1,, the majority of the datasets are single-label tasks, except GoEmotions (Demszky et al., 2020), Abstract and semeval201 8task1 which multi-label tasks.
Task
#Classes
Test size
Development
SST2
Sentiment, movie review Sentiment, movie review
2
1821
IMDB
2
25000
Student Question Catagories
Topic, subject of question Topic, news
4
5000
BBC News
5
1000
TREC-QC
Topic, questions
50
500
GoEmotons
Multi-label sentiment, emotions on Reddit
28
5427
Abstract
Multi-label topic, research articles
6
5243
CR
Sentiment, customer reviews electronics
2
376
SST5
Sentiment, movie reviews
5
2210
Testing
Emotion
Sentiment, emotions tweets
2000
AG News
Topic, news
7600
Yahoo Questions
Topic, questions Sentiment, reviews
10
5000
Yelp Review
5
5000
Semeval201 8task
Multi-label sentiment, affects in tweets (EN)
11
3259
Table
C Appendix: Anchor Statements
the experiments, the anchor statement is generated following the generic form
template + {w)
(1)
where w represent a describing word(s) of each particular class. In the multi-label cases, the anchor statements are generated per train sample by including all relevant class' descriptions
template + wi Wn
(2)
for = 1,,2,.,, .., n where n is number of clases in the sample. The listing below shows the template and class words for the different test tasks.
TEMPLATE =
SST5: "The movie is ()",
CR: "The movie is [}".
Emotion: "The emotion is ()",
AG News: "The topic is about ()",
Yelp Review: "The experience was ()",
Yahoo Questions: "The question is about O)",
SemEval2018Tace "The emotions are ()",

---

words= {
SST5: ["terrible", "bad", "okay", "good", "great"]. Emotion: [ ["saddess", " "oy""""" "ooy,"lov""" "loee" anger" "faar" "urpriss"".
CR: "bad"."good""
AG News: ["World", "Sports", "Business", "Sci/Tech"),
Yelp Review: [ ("terrible", "bad", "okay", "good", "great"], Yahoo Questions: | (SSocieyy & Culture",
"Science & Mathematies" "Health". "Education & Reference",
"Computers & Internet", "Sports", "Business & Finance", "Entertainment & Music", "Family & Relationships",
"Politics & Government"].
SemEval2018Taski: [' aanger', "anticipation', 'disgust,, 'fear', 'joy', 'love', 'optimism', 'pessimism" "sadness" 'surprise", trust']
Table 4 shows the anchor statement used in the ablation study described in Section 6 and corresponds the results shown in Figure 2.
Template
wilabels
wt letters
wt synonyms
SST2 IMDB
"The movie review is " "The movie review is
("negative', positive']
["A', 'B']
('bad', a 'good'] "bad'''good
I'negative', ''oositive' l'tech', "business', 'sport', 'entertainment', 'politics"]
['A', 'B"l
I'A', "B", 'C",
BBC News
"The topic is about
"D',E"] ['A', 'C', 'D'l
('Biology', 'Chemistry', 'Maths', 'Physics'1
SQC
"The subject is
Table 4: Template and words to generate anchor statement for different tasks in the ablation study. Note, the Permutation of labels is permutation of the labels shown under w:labels such that the class is assigned a wrong text label.
D Appendix: Implementation and Training details
SetFit We used the script implementation from SetFit code base?, and follow the settings in Tunstall et al. (2022). We experiment with the Sentence Transformer (Reimers and Gurevych, 2019) backbone model paraphrase-mpnte --aeee-bbbee2 available at HuggingFace (Wolf al., 2019) We generate pairs for fine-tuning by iterating 20 times over each train instance and pairing each with sample of the same class and one different class. This creates total of instances 220 pairs for fine-tuning. We fine-tune for epoch, batch size of 16, and max max-sequence length of 256, and use logistic regression as the classification head as in Tunstall et al. (2022).
AncSetFit AncSetFit is implemented based on SetFit implementation. For easier comparison, we keep the hyperparameters close as possible (see above). When generating train pairs, we iterate 40 times over the train instances, and for each instance, we construct the anchor statement for the particular class and sample an instance from different class. We set the iteration parameters to 40, such the total number of train pairs for fine-tuning becomes instances#4p pairs, in SetFit. The loss function's margin parameter c 0.25 for single-label tasks and e 0.5 for multi-label tasks, based on the development phase. The experiments are run with the anchor statements provided in Appendix C.
ADAPET We use the ADAPET method as modified in Tunstall et al. (2022) and implemented the SerFit code b The modification is that in these experiments, there is no access to development set,
https://gihubb.commmuugggggggggu https://huggigggaeecoooeeee httpp//pww...oooooooooooooooonee httpps//gggtabb.oommmggggggggggt

---

and therefore no way of choosing the best checkpoint, which is therefore disabled. The training runs for 1000 batches. Experiments are conducted both with the backbone model albert-xxlarge-v and with the smaller model albert-base-v2 7, For verbaliser and patterns, we use: "[TEXT] [template] [words]". where templates and words for the different tasks can be accessed in Appendix C.
We run PERFECT on the six different datasets using the codebase from SetFit fork from PERFECT codebase 9 with the standard hyperparameters and backbone model Roberta-large 10 (Liu et al., 2019) modify the code to add the datasets from Yahoo Questions and Yelp Review.
PERFECT
E Appendix: Running times
We compare running times, combining both training and inference times, for AncSetFit and ADAPET when training on 8 samples per class, running the script for one seed.
Time ADAPET
Time AncSetFit
CR
134 min
14 sce
SST5
150 min
22 sec
Emotion
136 min
23 sec
AG News
202 min
31 sec
The characteristics of the machine used are,
Intel Core i9 10940X 3.3GH: 14-Core MSI GeForce RTX 3090 2 STK 2 X 128GB RAM,
running Ubuntu 20.04.4 LTS.
"https:/huuginngfnce www.bee----lbbab.gg.e2
"ht http://tcttuuccmmmeesettt/eeette
"https: haatts::/huuginggaaae lar


Reviews:
[{"review_id": "KJDhb8Y7jw", "reviewer": "Reviewer_3p6y", "summary": "", "questions": "A) Why is ADAPET only reported on the table and not on the graphs?\n\nB) Why is the ablation data different from the data of the experimental results?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We like to thank you for taking the time to review our work and for the specific comments by which we can improve our work.  \nOur work aims at improving few-shot text classification in the cases where we both have extremely few data instances (2-8) and small model sizes, making the method resource light to run. Hence AncSetFit and SetFit can even be trained on a personal PC. This is why, we have focused mainly in the paper on comparing to SetFit (lines 231-235), as other methods use larger backbone models and more compute resources (see lines 270-274). ADAPET is getting better results in terms of accuracy in some of the tasks, but it also uses a backbone model with twice the number of parameters in the experiments (albert-xxlarge-v2), and a large factor more running time (see appendix, line 510-517). \n\nWe agree with your point, that our work could benefit from more comparisons. We are therefore suggesting to include the following in the paper:\n1) Results with the few-shot method PERFECT (Rabeeh et al. 2022)\n2) Results where ADAPET is using a backbone model (albert-base-v2)  which is equal in terms of model parameters as the backbone model used in AncSetFit and Setfit (ADAPET_base)\n3) Results with ADAPET (albert-xxlarge-v2) on the two  datasets Yahoo Questions and Yelp Review (ADAPET_xxlarge)\n\nWe would then also include a table showing the different number of model parameters used in each methods, such it easier to compare the methods both in terms of accuracy and sizes (see below).  We did not have time to run all the experiments in the author response period, but we have included the finished one here, all the new experiments is ran for 5 seeds, but we plan to run 10 seeds, we can see the following:\n\n-\tAncSetFit is outperforming PERFECT in all the tasks completed. \n-\tAncSetFit is outperforming ADAPET_base in all the tasks completed, showing that with an equal small model size, AncSetFit is superior \n-\tWe only had time to run ADAPET_xxlarge for size 2 for Yahoo Answers and Yelp review, and here AncSetFit is higher accurcay on Yahoo, but lower then ADAPET_xxlarge for Yelp.  \n\n\n| **Method**  | **Backbone**             | **Size** |\n|--------------------|---------------------------------|-----------------|\n|     PERFECT        |     roberta-large               |     355M        |\n|     ADAPET_xxl     |     albert-xxlarge-v2           |     223M        |\n|     ADAPET_base    |     albert-base-v2              |     117M        |\n|     SetFit         |     paraphrase-mpnet-base-v2    |     110M        |\n|     AncSetfit      |     paraphrase-mpnet-base-v2    |     110M        |\n<br />\n\n **Result table**\n\n|             | Method/size    | 2    | 4    | 8    |\n|-------------|----------------|------|------|------|\n| **Emotion**     | PERFECT        | 20.5 | 19.6 | 27.5 |\n|             | ADAPET_xxlarge | 46.7 | 48.1 | 53.9 |\n|             | ADAPET_base    | 37.5 | 43.6 | 49.3 |\n|             | SetFit         | 30.0 | 39.1 | 49.9 |\n|             | AncSetFit      | 49.1 | 53.2 | 56.7 |\n| -|\n| **AG News**     | PERFECT        | 49.5 | 69.6 | 80.0 |\n|             | ADAPET_xxlarge | 76.8 | 84.1 | 86.0 |\n|             | ADAPET_base    | 64.4 | 72.8 | 78.0 |\n|             | SetFit         | 65.3 | 76.5 | 82.8 |\n|             | AncSetFit      | 76.0 | 81.6 | 84.2 |\n| -|\n| **SST-5**      | PERFECT        | 25.8 | 32.0 | 36.2 |\n|             | ADAPET_xxlarge | 47.0 | 48.1 | 49.7 |\n|             | ADAPET_base    | 32.2 | 35.3 | 40.2 |\n|             | SetFit         | 33.4 | 38.4 | 42.3 |\n|             | AncSetFit      | 40.5 | 43.4 | 45.2 |\n| -|\n| **CR**          | PERFECT        | 62.2 | 76.6 | 81.1 |\n|             | ADAPET_xxlarge | 87.2 | 90.6 | 90.6 |\n|             | ADAPET_base    | 78.5 | 77.1 | 78.7 |\n|             | SetFit         | 75.5 | 86.8 | 89.6 |\n|             | AncSetFit      | 86.6 | 89.8 | 90.7 |\n|- |\n| **Yahoo Answers**       | ADAPET_xxlarge | 52.0 | p    | p    |\n|             | ADAPET_base    | 46.8 | 57.1 | 59.3 |\n|             | SetFit         | 42.9 | 55.4 | 61.0 |\n|             | AncSetFit      | 57.1 | 62.2 | 65.3 |\n| -|\n| **Yelp review** | ADAPET_xxlarge | 52.1 | p    | p    |\n|             | ADAPET_base    | 32.2 | 45.2 | p |\n|             | SetFit         | 36.6 | 44.9 | 50.7 |\n|             | AncSetFit      | 48.1 | 51.6 | 53.9 |\n\n*p: pending\n\n**Question A)**  On the graphs we compare SetFit and AncSetFit for training data sizes between 2-64. With the graphs, we show, that AncSetFit only has an advantage when we a dealing with really small training sizes (2-8). Our comparison with ADAPET therefore only focuses on data sizes between 2-8. We therefore did not plot it as we do not have results for the whole range. Comparing  ADAPET with SetFit for training data size of 64 can be seen in Tunstall et al. (2022). \n\n**Question B)** We wanted to follow a setup like in Tunstall et al. (2022), where we start by splitting a list of datasets into either development datasets or testing datasets. The idea is to do no hyperparameter-tuning on the test dataset in order to mimic a real few-shot settings as best as possible (See 167-170, 185-189 ). In the ablation study, we changed the templates and examined the effect of this. We did it on the development datasets, because we wanted to only have “one go” on the test datasets. We therefore also ran these experiments before we did any testing.\n\nWe do see the point, that this is not very clear in the text. We think it is a good idea to move the section to after the results and make it more clear.\n\n**References**\n\nRabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt-free and efficient few-shot learning with lan-\nguage models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin-\nguistics (Volume 1: Long Papers), pages 3638–3652,Dublin, Ireland. Association for Computational Linguistics.\n\nLewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke\nBates, Daniel Korat, Moshe Wasserblat, and Oren\nPereg. 2022. Efficient few-shot learning without\nprompts. arXiv preprint arXiv:2209.11055.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "AncSetFit improves the original work (SetFit) without being more computationally demanding.", "reasons_to_reject": "The experimental comparisons are lacking. The authors focuses on the comparison with SetFit but neglects most other methods. They only report ADAPET as comparison (and only for some of the datasets) which performs better than AncSetFit in 3 out of the 4 instances. \nMoreover, the paper of SetFit has more comparisons that in AncSetFit have not been considered at all.", "questions_to_authors": "A) Why is ADAPET only reported on the table and not on the graphs?\n\nB) Why is the ablation data different from the data of the experimental results?", "paper_topic_and_main_contributions": "The paper focuses on the low-data few-shot scenario in text classification. The authors extend a previous work (SetFit) by introducing semantic information for the labels, specifically they generate triplets with an \"anchor statement\" to incorporate the information into the loss. The proposed method (AncSetFit) is then tested on text classification datasets.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Consider moving the the ablation study after the results."}, "extra_text": {}}, {"review_id": "PAFjDzU7Kx", "reviewer": "Reviewer_qm4T", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We like to thank the reviewer for taking the time to review our work.\n\nOur work aims at improving few-shot text classification in the cases where we both have extremely few data instances (2-8) and small model sizes, making the method resource light to run. Hence AncSetFit and SetFit can even be trained on a personal PC. This is why, we have focused mainly in the paper on comparing to SetFit (lines 231-235), as other methods use larger backbone models and more compute resources (see lines 270-274). ADAPET is getting better results in terms of accuracy in some of the tasks, but it also uses a backbone model with twice the number of parameters in the experiments (albert-xxlarge-v2), and a large factor more running time (see appendix, line 510-517). \n\n We think we could make this point clearer in the paper by including more experiments where we compare to ADAPET with a similar backbone model size in terms of parameters as AncSetFit is using.\n We also propose to include more comparisons to another few-shot method. We propose to include the following,\n \n 1) Results on ADAPET using ‘albert-base-v2’, together with a table comparing the different model’s parameter sizes. \n 2) Results with the few-shot method PERFECT (Rabeeh et al. 2022)\n 3)Results with ADAPET (albert-xxlarge-v2) on the two datasets Yahoo Questions and Yelp Review (ADAPET_xxlarge)\n\nTime did not allow us to complete all the experiments, but below you can see the results we obtained so far, note we only had time to run 5 seeds for all the newly added experiments, but plan to run 10. We observe,\n\n-   AncSetFit is outperforming PERFECT in all the tasks completed. \n-   AncSetFit is outperforming ADAPET_base in all the tasks completed, showing that with an equal small model size, AncSetFit is superior \n-   We only had time to run ADAPET_xxlarge for size 2 for Yahoo Answers and Yelp review, and here AncSetFit is getting higher accuracy on Yahoo, but lower than ADAPET_xxlarge for Yelp.  \n\n| **Method**  | **Backbone**             | **Size** |\n|--------------------|---------------------------------|-----------------|\n|     PERFECT        |     roberta-large               |     355M        |\n|     ADAPET_xxl     |     albert-xxlarge-v2           |     223M        |\n|     ADAPET_base    |     albert-base-v2              |     117M        |\n|     SetFit         |     paraphrase-mpnet-base-v2    |     110M        |\n|     AncSetfit      |     paraphrase-mpnet-base-v2    |     110M        |\n\n\n\n **Result table**\n\n|             | Method/size    | 2    | 4    | 8    |\n|-------------|----------------|------|------|------|\n| **Emotion**     | PERFECT        | 20.5 | 19.6 | 27.5 |\n|             | ADAPET_xxlarge | 46.7 | 48.1 | 53.9 |\n|             | ADAPET_base    | 37.5 | 43.6 | 49.3 |\n|             | SetFit         | 30.0 | 39.1 | 49.9 |\n|             | AncSetFit      | 49.1 | 53.2 | 56.7 |\n| -|\n| **AG News**     | PERFECT        | 49.5 | 69.6 | 80.0 |\n|             | ADAPET_xxlarge | 76.8 | 84.1 | 86.0 |\n|             | ADAPET_base    | 64.4 | 72.8 | 78.0 |\n|             | SetFit         | 65.3 | 76.5 | 82.8 |\n|             | AncSetFit      | 76.0 | 81.6 | 84.2 |\n| -|\n| **SST-5**      | PERFECT        | 25.8 | 32.0 | 36.2 |\n|             | ADAPET_xxlarge | 47.0 | 48.1 | 49.7 |\n|             | ADAPET_base    | 32.2 | 35.3 | 40.2 |\n|             | SetFit         | 33.4 | 38.4 | 42.3 |\n|             | AncSetFit      | 40.5 | 43.4 | 45.2 |\n| -|\n| **CR**          | PERFECT        | 62.2 | 76.6 | 81.1 |\n|             | ADAPET_xxlarge | 87.2 | 90.6 | 90.6 |\n|             | ADAPET_base    | 78.5 | 77.1 | 78.7 |\n|             | SetFit         | 75.5 | 86.8 | 89.6 |\n|             | AncSetFit      | 86.6 | 89.8 | 90.7 |\n|- |\n| **Yahoo Answers**       | ADAPET_xxlarge | 52.0 | p    | p    |\n|             | ADAPET_base    | 46.8 | 57.1 | 59.3 |\n|             | SetFit         | 42.9 | 55.4 | 61.0 |\n|             | AncSetFit      | 57.1 | 62.2 | 65.3 |\n| -|\n| **Yelp review** | ADAPET_xxlarge | 52.1 | p    | p    |\n|             | ADAPET_base    | 32.2 | 45.2 | p |\n|             | SetFit         | 36.6 | 44.9 | 50.7 |\n|             | AncSetFit      | 48.1 | 51.6 | 53.9 |\n\n*p: pending\n\n**References**\n\nRabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt-free and efficient few-shot learning with lan-\nguage models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin-\nguistics (Volume 1: Long Papers), pages 3638–3652,Dublin, Ireland. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper proposes a simple but effective few-shot text classification method.", "reasons_to_reject": "- Compared with ADAPET, the performance of their proposed method is lower or on par, although their method has some merits over ADAPET.", "questions_to_authors": "", "paper_topic_and_main_contributions": "What this paper about? \n- This paper tackles Few-Shot Text Classification. \n\nWhat problem or question did this paper address?\n- The first research question (RQ 1) is if the efficiency of small models in the SetFit method can be retained while also making the method label-aware and if this improves performance, especially in cases with little training data (Line 31-35).\n- The second research question (RQ 2) is to what extent variation in textual information affects model performance (Line 40-41).\n\nWhat are the main contributions that this paper makes towards a solution or answer?\n- For RQ 1: In order to inform the model of what is to be classified, the authors proposes to add semantic information about the task and labels.\n- For RQ 2: The authors used four variations of constructing anchor statement and demonstrated the impacts of the variations on the performances (Figure 1).\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "3ZUQ9HOQwQ", "reviewer": "Reviewer_Dzxn", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We like to thank the reviewer for taking the time to review our work.\n\nOur work aims at improving few-shot text classification in the cases where we both have extremely few data instances (2-8) and small model sizes, making the method resource light to run. Hence AncSetFit and SetFit can even be trained on a personal PC. This is why, we have focused mainly in the paper on comparing to SetFit (lines 231-235), as other methods use larger backbone models and more compute resources (see lines 270-274). ADAPET is getting better results in terms of accuracy in some of the tasks, but it also uses a backbone model with twice the number of parameters in the experiments (albert-xxlarge-v2), and a large factor more running time (see appendix, line 510-517). \n\n We think we could make this point clearer in the paper by including more experiments where we compare to ADAPET with a similar backbone model size in terms of parameters as AncSetFit is using.\n We also propose to include more comparisons to another few-shot method. We propose to include the following,\n \n 1) Results on ADAPET using ‘albert-base-v2’, together with a table comparing the different model’s parameter sizes. \n 2) Results with the few-shot method PERFECT (Rabeeh et al. 2022)\n 3) Results with ADAPET (albert-xxlarge-v2) on the two datasets Yahoo Questions and Yelp Review (ADAPET_xxlarge)\n\nTime did not allow us to complete all the experiments, but below you can see the results we obtained so far, note we only had time to run 5 seeds for all the newly added experiments, but plan to run 10. We observe,\n\n-   AncSetFit is outperforming PERFECT in all the tasks completed. \n-   AncSetFit is outperforming ADAPET_base in all the tasks completed, showing that with an equal small model size, AncSetFit is superior \n-   We only had time to run ADAPET_xxlarge for size 2 for Yahoo Answers and Yelp review, and here AncSetFit is getting higher accuracy on Yahoo, but lower than ADAPET_xxlarge for Yelp.  \n\n| **Method**  | **Backbone**             | **Size** |\n|--------------------|---------------------------------|-----------------|\n|     PERFECT        |     roberta-large               |     355M        |\n|     ADAPET_xxl     |     albert-xxlarge-v2           |     223M        |\n|     ADAPET_base    |     albert-base-v2              |     117M        |\n|     SetFit         |     paraphrase-mpnet-base-v2    |     110M        |\n|     AncSetfit      |     paraphrase-mpnet-base-v2    |     110M        |\n\n\n\n **Result table**\n\n|             | Method/size    | 2    | 4    | 8    |\n|-------------|----------------|------|------|------|\n| **Emotion**     | PERFECT        | 20.5 | 19.6 | 27.5 |\n|             | ADAPET_xxlarge | 46.7 | 48.1 | 53.9 |\n|             | ADAPET_base    | 37.5 | 43.6 | 49.3 |\n|             | SetFit         | 30.0 | 39.1 | 49.9 |\n|             | AncSetFit      | 49.1 | 53.2 | 56.7 |\n| -|\n| **AG News**     | PERFECT        | 49.5 | 69.6 | 80.0 |\n|             | ADAPET_xxlarge | 76.8 | 84.1 | 86.0 |\n|             | ADAPET_base    | 64.4 | 72.8 | 78.0 |\n|             | SetFit         | 65.3 | 76.5 | 82.8 |\n|             | AncSetFit      | 76.0 | 81.6 | 84.2 |\n| -|\n| **SST-5**      | PERFECT        | 25.8 | 32.0 | 36.2 |\n|             | ADAPET_xxlarge | 47.0 | 48.1 | 49.7 |\n|             | ADAPET_base    | 32.2 | 35.3 | 40.2 |\n|             | SetFit         | 33.4 | 38.4 | 42.3 |\n|             | AncSetFit      | 40.5 | 43.4 | 45.2 |\n| -|\n| **CR**          | PERFECT        | 62.2 | 76.6 | 81.1 |\n|             | ADAPET_xxlarge | 87.2 | 90.6 | 90.6 |\n|             | ADAPET_base    | 78.5 | 77.1 | 78.7 |\n|             | SetFit         | 75.5 | 86.8 | 89.6 |\n|             | AncSetFit      | 86.6 | 89.8 | 90.7 |\n|- |\n| **Yahoo Answers**       | ADAPET_xxlarge | 52.0 | p    | p    |\n|             | ADAPET_base    | 46.8 | 57.1 | 59.3 |\n|             | SetFit         | 42.9 | 55.4 | 61.0 |\n|             | AncSetFit      | 57.1 | 62.2 | 65.3 |\n| -|\n| **Yelp review** | ADAPET_xxlarge | 52.1 | p    | p    |\n|             | ADAPET_base    | 32.2 | 45.2 | p |\n|             | SetFit         | 36.6 | 44.9 | 50.7 |\n|             | AncSetFit      | 48.1 | 51.6 | 53.9 |\n\n*p: pending\n\n**References**\n\nRabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt-free and efficient few-shot learning with lan-\nguage models. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin-\nguistics (Volume 1: Long Papers), pages 3638–3652,Dublin, Ireland. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work utilizes semantic label Information and improves the performance of SetFit.", "reasons_to_reject": "This work aims to improve SetFit but compared with experimental design in SetFit, existing results in this work fail to demonstrate AncSetFit effectiveness.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper addressed semantic information of SetFit, which aims to improve the performance in extremely few-shot scenario.", "excitement": 2, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces AncSetFit, a method for improving few-shot text classification in resource-constrained scenarios. It incorporates semantic label information and focuses on scenarios with minimal training data and small model sizes, aiming for efficiency. The authors conducted experiments on various datasets, comparing AncSetFit to SetFit and ADAPET. Reviewers appreciated the simplicity and efficiency of the proposed method but raised concerns about limited comparisons and the performance relative to ADAPET. While the paper's contributions are smaller due to it being a short-paper, the authors have taken steps to address the reviewers' concerns and enhance the comprehensiveness of their work.

**Key Concerns of Reviewers:**

Reviewers expressed concerns about the paper's limited experimental scope, focusing mainly on comparisons with SetFit and not including comparisons with other methods.
There were concerns about the performance comparison with ADAPET, which outperformed AncSetFit in some instances, possibly due to differences in model size and computational resources.
The authors responded by proposing additional experiments and comparisons, including results with ADAPET using a smaller backbone model and introducing results with the PERFECT method. 

The authors are encouraged to extend the abstract with 1-2 sentences that briefly describe the proposed method in more detail.

**Note:** The review and scores of reviewer "@Dzxn" need to be ignored due to the low-quality review.
Decision: Accept-Main