AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification
Authors: Yongxin Huang
Kexin Wang
Sourav Dutta
Raj Nath Patel
Goran Glavaš
Iryna Gurevych
Venue: EMNLP 2023 Main (2023)
Keywords: few-shot text classification
sentence embedding
domain adaptation
parameter-efficient fine-tuning
TL;DR: We efficiently derive domain-adapted sentence embeddings for few-shot classification.

Abstract: Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) is efficient, robust, and effective. In this work, we investigate strategies for domain-specialization in the context of few-shot sentence classification with SEs. 
We first establish that unsupervised Domain-Adaptive Pre-Training (DAPT) of a base Pre-trained Language Model (PLM) (i.e., not an SE) substantially improves the accuracy of few-shot sentence classification by up to 8.4 points. However, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domain) Sentence Embedding Pre-Training (SEPT). On the other hand, applying general-domain SEPT on top of a domain-adapted base PLM (i.e., after DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top of a DAPT-ed PLM of each domain. As a solution, we propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent's effectiveness in extensive experiments on 17 different few-shot sentence classification datasets. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs. The code for AdaSent is available.

AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification
Yongxin Huang', Kexin Wang', Souray Dutta",
Raj Nath Patel?, Goran Glavaš Iryna Gurevych
!Ubiquitous Knowledge Processing Lab (UKP Lab)
Department of Computer Scienee and Hessian Center for Al (hessian.Al)
Technical University of Darmstadt
Huawei Research Centre, Dublin, Ireland
Center for AI and Data Science, University of Würzburg
www. ukp. tu-darmstadt de
2 [sourav. dutta2, raj. nath. patel wehuawei. com
3 goran an. www.seburrwwerrrburg........
Abstract
Recent work has found that few-shot sentence classification based on pre-trained Sentence Encoders (SEs) efficient, robust, and effec- tive. In this work, investigate strategies for domain- Nomain-speecallie specialization in the context of few- shot sentence classification with SEs. We first establish that unsupervised Domain-Adaptive Pre- Tre-Traning (DAPT) a base Pre-trained Lan- guage Model (PLM) (i.e., not an SE) substan- tially improves the accuracy of few-shot sen- tence classification by up 8.4 points. How- ever, applying DAPT on SEs, on the one hand, disrupts the effects of their (general-domall Sentence Embedding Pre- Tre-Trainnn (SEPT). On the other hand, applying seneral-domain SEPT on top of domain-adaan adaated base PLM (i.c., af- DAPT) is effective but inefficient, since the computationally expensive SEPT needs to be executed on top DAPT-ed PLM each domain. As solution, propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on base PLM. The adapter can be inserted into DAPT-ed PLMs from any domain. We demonstrate AdaSent's effective- ness in extensive experiments 17 differ- ent few-shot sentence classification datasets. AdaSent matches or surpasses the performance full SEPT DAPT-ed PLM, while substan- tially reducing the training costs. The code for AdaSent is available!
PLM
PLM
A Adptter
DAPT
SEPT
on task spec-sp unlabeled data
on paraplurae data generall domain
PLM
A depter
SetFit
few-shot labeled data
Figure I: Training diagram of AdaSent. Trainable pa- rameters are marked in green. After Domain -AdaiieA Pre-training (DAPT) on the Pre-Trained Language Model (PLM) and Centence-Embeddding Pre-Training (SEPT) with an adapter, the two parts are assembled together perform SetFit for few-shot classification.
pre-trained sentence embeddings. Being prompt- free and effective on relative small models, Set- Fit is much more efficient than popular prompt- based methods including In-Context Learning (ICL, Brown et al., 2020) and Pattern Exploit Training (PET, Schick and Schütze, 2021), which require careful prompt engineering and large model size.
Despite its success, SetFit fine-tunes a sentence encoder with only few labeled samples without leveraging unlabeled data from the target-task do- main, which are easy to obtain. It is well-known that Domain-Adaptive Pre-Training (DAPT)² on vanilla PLM with unlabeled in-domain data can significantly improve its downstream perfor- mance (Han and Eisenstein, 2019; Gururangan et al., 2020). However, it is ineffective to apply
1 Introduction
Few-shot learning aims at training an effective model with a few labeled examples, reducing the cost of developing models for new domains and tasks. In recent work. SetFit (Tunstall et al., 2022) achieves strong performance in few-shot classifica- tion by contrastively fine-tuning (Koch al., 2015)
By DAPT we refer to the TAPT (Task-Adaptive Pre- Training) in Gururangan et al. (2020). We do not strictly differentiate between domain and task in the present work.
"https://githen" LoopaaaaAccc/KLLa/aAAASeeee

---

DAPT on sentence encoders, i.e. vanilla PLMs that have undergone Sentence Embedding Pre-Training (SEPT, Reimers and Gurevych, 2019) in general domain, as DAPT messes up the effects of SEPT and disrupts the model's ability to semantically ac- curately embed sentences. Though DAPT before SEPT is effective in contrast (Wang et al., 2021), it is computationally inefficient as the general- domain SEPT has to be done all over again on every domain-adapted PLM if we have more than one domain.
ing (Reimers and Gurevych, 2019), their usage is less explored in text classification. Though frozen sentence embeddings can directly serve as input features in text classification (Perone et al., 2018; Piao, 2021), the performance limited compared to standard full fine-tuning of PLMs (Kumar et al., 2022). To compensate this performance loss, Patel et al. (2021) concatenate encodings from various Sentence Transformers to form semantically richer sentence representations, achieving results com- parable to standard fine-tuning, but the cost of slower inference. More recently, SetFit (Tunstall et al., 2022) significantly improves the few-shot classification by contrastively fine-tuning a pre- trained sentence-embedding model before training classification head. Despite efficiently utilizing the limited labeled samples, SetFit does not lever- age the abundant in-domain unlabeled data that can provide more domain knowledge for the task.
To create a Nomain-sppcialinn sentence en- coder for few-shot sentence classification both effi- ciently and effectively, we propose AdaSent, which combines DAPT and SEPT in modular fashion. Specifically, it stores the sentence-specializatie abilities obtained via single SEPT procedure in the general domain into an adapter. This sentence- encoding adapter is trained once regardless of the number of domains, and can be plugged into domain-adapted PLMs from various domains to make them domain-specializen sentence encoders, on which SetFit is carried out to do downstream classification training (Figure 1). Our experiments show that AdaSent can match or surpass the in- efficient "full SEPT after DAPT" approach's 's per- formance on 17 sentence classification tasks from various domains. The contribution of AdaSent is two-fold:
2.2 Few-Shot Text Classification
Large language models can perform few-shot clas- sification through ICL with task-specific prompts consisting of a few labeled examples (Brown et al., 2020). Though it avoids any gradient update, ICL relies on large model sizes for good performance, which makes inference costly. Prompt-based fine- tuning, the other hand, can work with smaller models (Schick and Schüitze, 2021; Tam al., 2021; Gao al., 2021a). Parameter Efficient Fine- Tuning (PEFT) can further reduce the training cost by fine- mac-tuning& a much smaller module in a frozen PLM (Houlsby et al.. 2019; Li and Liang. 2021; Hu et al.. 2022; Karimi Mahabadi et al., 2022; He et al., 2022; Liu et al., 2022). As an alternative way to employ task instructions, Su et al. (2023) train domain- and task-aware text embeddings by prepending instructions to the input text. In con- trast to these methods, SetFit and our approach not only require a smaller model size, but also elim- inate the need for prompts or instructions, which can introduce large variance and should be care- fully designed (Perez et al., 2021).
AdaSent significantly improves SetFit, the previous state-of-the-art few-shot classifica- tion approach, by leveraging unlabeled task- specific data through DAPT.
AdaSent resolves the conflict between DAPT and SEPT and the efficiency issue of the se- quential execution of both training procedures, by combining them in a modular fashion with- out sacrificing the performance.
2 Related Work
2.1 Text Classification with Sentence Embeddings
2.3 Domain Adaptation of Language Models
Transformer-basee (Vaswani et al., 2017) Pre- trained Language Models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Sanh et al., 2019) can be fine-tuned build sentence embedding mod- els (Reimers and Gurevych, 2019). Since the orig- inal goal of training sentence embeddings is to better model the sentence similarity for applica- tions such as dense retrieval and sentence cluster-
One typical way for creating domain-specinn lan- guage models is pre-training through Masked Lan- guage Modelling on in-domain corpora, either con- tinuously (Gururangan et al., 2020) or from-scratch (Lee et al., 2019). An alternative is adapting the tokenizer to accommodate domain speciffc vocab- ulary (Sachidananda et al., 2021; Yao et al., 2021).

---

For sentence embedding models specifically, do- main adaptation is usually done through unsuper- vised training with novel objectives (Wang al., 2021; Liu and Yang. 2022) or in-domain data gener- ation (Wang et al., 2022), mainly for the similarity or relevance estimation tasks. However, supervised sentence embedding training with general- -omain data (SEPT) is always required after the unsuper- vised domain-specific training phase (DAPT) to achieve optimal performance (Wang et al., 2021). Our proposed method is inspired by the idea of dis- entangling domain adaptation and the downstream relevance estimation task via PEFT in Zhan et al. (2022). In the present study, we show that PEFT can also be used to decouple DAPT and SEPT for few-shot classification tasks.
where u,v RD are the D-dimensional sen- tence embeddings of two sentences respectively and y/ E {0, 1] is the pair label. This aims to push instances of the same classes closer together in the representation space and those from different classes further apart, thereby clustering sentences according to their class labels to provide clearer decision boundary for the classifier training later. In the second step, the Transformer is frozen embed the original few-shot sentences. These sen- tence embeddings are used as input features to train simple Logistic Regression (Cox, 1958) classifi- cation head.
3.2 Sentence Embedding Pre-Training (SEPT)
As will be shown in subsection 6.2, the success of SetFit heavily relies on SEPT. This is because the averaged word representations or the [CLS] representation from PLM cannot capture the sen- tence semanties well without further training with sentence-level objectives (Reimers and Gurevych, 2019). The purpose of centence-embediee training is to train universal semantic representa- tions that can be fine-tuned for different down- stream tasks, e.g. in SetFit. Unlike SetFit, sen- tences with similar meaning are brought closer to- gether in SEPT, while those with dissimilar mean- ings are pushed apart. Sentence pairs for this kind of contrastive training are typically obtained from Natural Language Inference (NLI, Bowman et al., 2015; Williams et al., 2018) or paraphrase datasets in the general domain. Sentence pairs labeled as "entailment" or "paraphrase" the origi- nal datasets are used as positive pairs, i.c. sentences with similar meaning, in SEPT. The Multiple- Negative Ranking Loss (MNRL, Henderson et al.. 2017) with in-batch negatives is usually applied for training:
2.4 Semi-Supervised Text Classification
Unsupervised data can be incorporated in various ways improve few-shot classification. While the DAPT approaches in subsection 2.3 allow the model to learn Nomainn-spccinn features in a task- agnostic way, other semi-supervised methods typi- cally propagate task information from labeled data to unlabeled data through pseudo labeling. The pscudo-labelee data either used for self-training (Schick and Schütze, 2021) consistency training (Xie et al., 2020). All these approaches can also be combined to enable more efficient use of unlabeled data (Li et al., b021b; Chen et al., 2021; Zhao and Yao, 2022). our experiments, we found that sim- ple self-training using the same data for DAPT can further improve the performance of AdaSent.
pre-
3 Background
3.1 SetFit
SetFit (Tunstall et al., 2022) is a two-step train- ing procedure based on pre- pre-trained sentence- embedding Transformer models for few-shot sen- tence classification. In the sentence-embedding fine-tuning step, positive and negative sentence pairs are generated from few-shot labeled sentences as follows: Pairs consisting of sentences from the same class are labeled positively with a score of and pairs of sentences from different classes are assigned negative score of O. These generated pairs are used to fine-tune the sentence-embedding model with the Cosine Similarity Loss:
1 K
cos (oomssimmaass
LMNRI
log
coos sim
where {1;, wi) V= are batch of K positive sen- tence pairs.
3.3 Domain-Adapted Sentence Embeddings
The definition of sentence similarity varies from domain to domain, but labeled data for SEPT are usually expensive to obtain in specialized domains. Wang et (2021) found that comain-adapted sen- tence embedding models can be trained following a two-stage recipe: first doing unsupervised DAPT
Ligesine = -ly cos_simm(a)) v)

---

Basc PLM
Base PLM
Basc PLM
Basc
Base PLM
Transformer
PLM
dapleer
SEPT
Trained DAPT
Trained with
SEPT
SEPT
DAPT
DAPT
Shared across tasks
(1) SEPT
(2) DAPT= (AdaSent)
(3)
(4)
(5) SEPT-DAPPas
SEPT-DAPT
Figure 2: Five ways to combine Domain-Adaptivv Pre- Traininnn (DAPT) and Sentence Embedding Pre-Training (SEPT). An arrow pointing from a Transformer to an adapter means the adapter is trained on that Transformer. A dashed line means simple module assembly without any parameter tuning. e marks trained parameters that are reusable and shared across downstream tasks. In contrast, all SEPT training starting from DAPT Transformer (red arrows) must be repeated on every downstream task.
(e.g. MLM) on the domain-specific corpus, then applying supervised SEPT in the general domain (Figure (1)). With this training order, if we want train models for various domains, the same sec- ond stage has be repeated for every domain, al- though it does not involve any domain-specific data. Such computational overhead cannot be avoided by simply reversing the order of the two training stages (Figure 2 (4)), since it has been shown in previous work that DAPT after the generic sentence embedding training has negative impact the downstream performance (Wang al., 2021).
Data
All
Paraphrase
NLI+SC+SE
NLI
Data Size
1B
86M
0.6M
0.3M
Accuracy
68.6
70.0
70.0
68.8
Table SetFit accuracy on the MTEB classification tasks (see subsection 5.3) of sentence embedding mod- els trained on different SEPT datasets without do- main adaption. All and Paraphrase stand for the all- distilroberta-vl and the paraphrase-diste distilroberto base- v24, respectively.
reasons. Most significantly, SEPT adapter is trained only once and shared across various down- stream classification tasks, avoiding the overhead of repeating SEPT on new DAPT-ed models. More- over, AdaSent allows for the independent execu- tion of DAPT and SEPT, eliminating the need for sequential training. Therefore, they can be run con- currently in parallel to save training time. Lastly, training an adapter instead of the full model in SEPT reduces the number of trainable parameters.
4 Method
As illustrated in Figure our method for few-shot classification with domain-adapted sentence em- beddings consists of three parts of training: (I) DAPT on the base PLM with task-specific unla- beled data, (2) SEPT on an adapter module with labeled sentence pairs from the general domain and (3) SetFit the whole architecture (i.e. both the PLM and the adapter) with few-shot labeled data. In the first part, specifically, continue train a base PLM like Bostil on unlabeled tar- get task data with the MLM loss to learn domain- specific language knowledge. In another separate procedure, SEPT done by tuning an adapter on frozen base Transformer (the same PLM as in DAPT) without any domain adaptation. Once the domain- domain-independent sentence encoding adapter is trained, it can be easily inserted into different DAPT models, ready for the few-shot classification task learning via SetFit in the third part.
Given the extensive number of experiments this study, we use a mixture of three datasets for SEPT, dubbed NLI+S++SE, consisting SNLI (Bowman et al., 2015) + MultiNLI (Williams et al., 2018), Sentence Compression (Filippova and Al- tun, 2013) and StackExchange duplicate questions, for the sake of simplicity. This is a much smaller subset of the billion sentence pairs which the popular off-the-shelf sentence embedding models are pre-trained. We found that these three SEPT datasets transfer the best for the downstream clas-
hhttps://huggiigfaaee wtt..//eeeeeeeeeeeeeeteoooo http:////eeeeeeeeeeeoooooooottar all ---dsiilrrbberaaaiii
hnttps://hugginggacc htt.://eeeoeeeeeeeeeerararrrreee tratreettrrrnfffrrsssss perphhrase-dittiirobert---bba
Compared to the previous approach described in subsection 3.3, AdaSent is more efficient for three
"https://hugginggaace co/ sentencce centence-transforeeee transformers

---

sification tasks and are adequate to train model that performs on par with or even better than off- the-shelf models as shown in Table 1.
full and parameter-efficient training, respectively, For parameter-effiiiet training, a parallel adapter (He et al., 2022) is used by default. We also provide results of three other different PEFT methods: bot- tleneck adapter (Houlsby et al., 2019; Pfeiffer et al., 2020), LoRA (Hu et al., 2022) and prefix-tuning (Li and Liang, 2021).
5 Experimental Setup
5.1 Models
We experiment with three baselines and five types of domain-adapted sentence embedding models. All of these models serve as the sentence encoder in the SetFit for the few-shot classification tasks. The baselines are: (1) Base, the base PLM without any DAPT or SEPT; (2) SEPT, with only SEPT on the base PLM, which is also the default en- coder in the original SetFit work; (3) DAPT, Nomain-adaptee PLM, i.c. the Base model continu- ously pre-trained the in-domain corpus with- out SEPT. We also experiment with five varia- tions of domain-adapted sentence embedding mod- els, which differ in the way SEPT and DAPT are combined (Figure 2). In detail, they are: (1) created through DAPT followed by SEPT on the full Transformer parameters with- out adapter; (2) DAPT+SEPT is our AdaSent model; (3) DAPT-SEPT,.a differs from AdaSent in the training of the SEPT adapter, which is trained on the DAPT model instead of the base PLM; (4) SEPT SEPT-DDAPT reverses the training or- der of (1), namely doing DAPT after SEPT: (5) trains a DAPT adapter on a frozen SEPT model. It requires the shortest training time, since it avoids any update of the Transformer parameters.
In separate experiment (subsection 6.1), we compare, on models DAPT, DAPT-SEPT and SEPT->DAPT, three objectives for DAPT: MLM, TSDAE (Wang et al., 2021) and SimCSE (Gao et al., 2021b). The latter two are designed for unsu- pervised sentence embedding learning, represent- ing two mainstream training objectives for this task: denoising autoencoding and contrastive learning, respectively. For all three objectives, the unlabeled dataset for 3 epochs. The batch sizes are 8, 64 and the learning rates are 5e-5, 3e-5 and le-2, respectively. We only use NLJ data in SEPT here for simplicity. The same setting is applied for the experiment in subsection 6.5.
train on
For each downstream classification task, we do SetFit on all the models with 8-shot labeled data per class for epoch. The default classification head in Set is Logistic Regression.
5.3 Evaluation
We evaluate the models on 17 classification tasks, an overview of which is provided in Table 2. These include 11 datasets from the MTEB (Massive Text Embedding Benchmark, Muennighoff et al., 2023). For datasets that contain multilingual data, we only use the English subset in this work. Since most of the MTEB tasks are from the general domain, we add another six tasks for Nomain-specciie cases, in- cluding Adverse Drug Events Binary Classification (Gurulingappa et al., 2012) and PubMed RCT (Der- noncourt and Lee, 2017) from the biomedical do- main, LEDGAR (Tuggener et al., 2020; Chalkidis et al., 2022) from the legal domain, as well as Fi- nancial PhraseBank (Malo al., 2014), Twitter Financial News Sentiment" and Twitter Financial News Topic from the financial domain.
5.2 Training Details
We use Dostil as the base PLM in our main experiments. Additional results on Distil- BERT are reported in the Appendix D. We set the maximum sequence length to 512. We do not tune the hyperparameters and keep them the same for all downstream tasks. If not stated otherwise. the de- fault setting in the used libraries (cf. Appendix A) is applied. For DAPT with MLM in the main exper- iments, we train for a fixed number of 2344 steps? with batch size of 256. When using PEFT meth- ods for DAPT, we keep the same batch size and number of steps, but with larger learning rate of le 4. For SEPT, we train with a batch size of 64 for epoch; the learning rates are 2e-5 and le-4 for
For each task, we sample 8 shots per class from the training set as the labeled data for SetFit and treat the whole original training set as the unla- beled data for DAPT. We run SetFit five times with different random seeds, which correspond to five
"nttps://huggingfacce http://ggggggaae..coo///ttsss//e cotttttttttttte/eeeeettt.tt wwittter-finnnncal--nwwsss-sss semtiment
*Sce Appendix B for results of individual SEPT datasets. 'This corresponds to 3 epochs on the largest training set in our evaluation datasets.
"nttps://hugginggaace co/ ///daaastts/eershot/// twiitter-finaniial-nee -eweiialllleeweooooeo topic

---

Seq. len.
Dataset
Abbr.
# Train
Class
Description
(words)
MTEB classificatinn
Amazon Counterfactual
AC
4018
2
20
Amazon customer reviews labeled counterfactual
Banking77
BANK
10003
77
Banking querys with corresponding intents.
Amazon Massive Intent
AMI
11514
60
Amazon Alexa ulterances with associated intent.
Amazon Massive Scenario
AMS
11514
18
6
Amazon Alexa ulterances with theme.
MTOP Intent
MI
15667
113
Task-oriented dialog utterances with intent. ursk-eri dialog ullerances with domain.
MTOP Domain
MD
15667
11
Emnttio
EMO
16000
6
19
Twitter messages with basic emotion type.
IMDb
IMDB
25000
2
233
Movic reviews positive or negative.
Twitter Sentiment Extraction
TSE
27481
3
12
Tweet sentiment classification as neutral, positive negative.
Toxic Conversation
TC
50000
2
51
Comments from the Civil Comments platform toxic not.
Amazon Reviews Multi
ARM
200000
5
38
Amazon reviews with 1-5 stars.
Domainn-speciie tasks
Pinancial PhraseBank
TPB
3876
3
23
Financial news headlines with the view of a retail investor.
Twitter Financial News Sentiment
TENS
8588
3
12
Finance-related tweess with their sentiment.
Twitter Financial News Topic
TENT
15291
20
18
Finance-related twezts with their topic.
Adverse Drug Events
ADE
18812
2
19
Classify if a sentence is ADE-related or not.
PubMed RCT
RCT
176642
5
27
PubMed abstract sentences with their role in the abstract.
LEDGAR
LED
60000
100
114
Contract provisions with their main topic.
Table 2: Overview of the evaluation datasets. All tasks multi-class classification. From the training only 8 labeled shots per class are used for SetFit. The whole training set is used in DAPT without labels. Examples from each dataset can be found in Appendix F.
70
it may distort the sentence representation space shaped by SEPT. These findings on our classifica- tion tasks are consistent with those on the retrieval tasks in Wang et al. (2021).
Base SEPT
68-
DAPT
Accuracy
66
SAPT-DDPP
DAPT SEPT
With the DAPT-SEPT order, MLM achieves the best result among three DAPT objectives, im- proving the SEPT baseline by around 3 points on average. Although TSDAE has been shown to have clear advantage in tasks like re-ranking and para- phrase identification (Wang al., 2021), 1), it out to be suboptimal for sentence classification. On the contrary. MLM performs worse than TSDAE and SimCSE when there is no SEPT. We suppose that sentence classification with SetFit requires a good representation of both token- and sentence- level semantics, which are learned through MLL and SEPT respectively in the DAPTMIM setting. In other settings, either supervised sen- tence embedding training is absent (only DAPT), or token representation learning is missing (both TSDAE and SimCSE are for sentence representa- tion learning).
60
58
SimCSE
TSDAE
MLM
Figure 3: Averaged accuracy on datasets of different DAPT training objectives (SimCSE, TSDAE, MLM) and different training strategies (without, before or after SEPT). Results individual datasets are in Table 11.
different sets of few-shot samples. We report the average accuracy on the test set of each dataset over the five runs.
6 Results
6.1
Training Order and DAPT Objectives
In our first experiment, we compare two training or- ders: SEPT-DDAPT and SEPT- and three DAPT objectives: MLM, TSDAE and SimCSE. The results are shown in Figure 3.
6.2 Combination of DAPT and SEPT
Regarding the training order, HAPT~>SERS out- performs SEPT~>AAPT for all three DAPT objec- tives. DAPT can enhance the SEPT baseline only when performed prior SEPT, but this setting has the efficiency issue described subsection 3.3. On the other hand, DAPT has a negative impact on an already pre-trdi sentence encoder, because
In this subsection, we present the results of our main experiments various combination strate- gies for DAPT and SEPT. The results on the MTEB datasets are reported Table 3, and those for the domain-specific datasets are in Table AdaSent achieves the best result on 10 out of 17 tasks, out- performing the not domain- adapted SEPT model

---

Row
Model
AC
BANK
AMI
AMS
MD
EMO
IMDB
TSE
TC
ARM
Avg.
No.
No SEPT
RI
Base
65.9
75.7
62.0
71.0
72.8
89.4
40.2
67.7
50.9
55.2
37.3
62.6
R2
DAPT
80.4
70.0
94.7
37.3
74.9
55.1
48.2
41.9
66.5
Full SEPT
R3
SEPT (prev. SOTA)
76.1
77.0
66.8
73.3
78.4
90.6
52.2
84.8
63.2
63.6
44.2
70.0
R4
DSPT-SEEP
73.8
80.9
73.5
79.2
83.7
94.5
54.0
86.2
63.7
63.6
46.9
72.7
SEPT on Adapter
R5
&
76.0
76.9
79.4
91.4
55.3
84.6
63.1
43.8
70.5
R6
DAPT-
77.9
80.7
73.8
79.3
82.9
94.7
54.7
85.6
65.0
65.5
47.0
73.4
R7
AdaSent
77.9
80.61
73.7
80.51
82.71
95.4
54.1
86.7
65.2
63.0
48.1'
73.4
DAPT Adapter
R8
HEPT-DAPTAL
72.8
79.5
69.8
78.0
81.0
93.7
48.3
84.4
59.2
59.4
44.9
70.1
Table Classification accuracy on the MTEB classification tasks. Full SEPT means tuning all the PLM parameters in the sentence embedding pre-training. Best results on each dataset are in bold. i marks the cases where AdaSent outperforms SEPT (R5) with statistical significance level of 0.05.
Row
Model
FPB TENS TENT ADE RCT
LED Avg.
No SEPT
70-
Base
49.2
60.7
49.6
64.2
55.4
68
Bottlonack adapter
DAPT
50.3
56.3
64.8
67.8
57.8
66.7
60.6
Paraliet adapter
Full SEPT
66
Prefa-tating
62.3
61.5
65.6
64
R4
DAPT-SEPT
65.6
69.4 68.4
67.4
66.5
68.1
67.6
SEPT Adapter
62-
SEPT
64.2
66.1
61.4
62.8
58.7
65.9
63.2
60-
DAPT SSPT-SSEPTAL mns
66.1
69.9
68.5
65.8
67.4
68.0
67.6
AdaSent
66.4
69.8
68.6
67.8
67.5
67.8
68.0
DAFT+ SEPT - (AdaSeat)
DAPT Adopter
R8
SEPT--DAPTsh
62.7 62.0
65.2
66.7
64.0 66.8 64.6
Figure 4: Averaged accuracy different PEFT methods. SEPT PEFT stands for SEPT PEFT module. More detailed results are available in Table 13.
Table 4: Classification accuracy the domain-specific datasets. Best results on each dataset are bold. f marks the where AdaSent outperforms SEPT (RS) with statistical significance level of 0.05.
across different downstream tasks, while obtain- ing comparable results without statistically signif- icant difference (R7 vs. R4), except for the AMS dataset, where Adasent is even significantly better than DAPT The comparable performance and AdaSent (R6 R7)
by 3.9 on average on the MTEB datasets, and more prominently, by 4.7 on the datasets in Table 4 with a larger domain shift from the pre-training data. The improvement is statistically significant on 8 datasets, with significance level of 0.05. Our fol- lowing analysis will focus on Table while similar trends can be observed in Table 4.
of DAPT->SEPE proves the viability of decoupling DAPT and SEPT: The SEPT adapter does have to be trained on specific DAPT model. Instead of doing SEPT on adapter, we also tried with DAPT on adapter SEETT>DAPTT.dh which should be the most effi- cient method explained in subsection 5. Disap- pointingly. it can barely improve over SEPT (R8 vs. R3) and is much worse than AdaSent (R8 vs. R7). The reason could be that this setting suffers from the same problem SEPT-DAPP, as the DAPT phase, despite on an adapter, still conducted after SEPT.
SEPT crucial to the final accuracy of classi- fication methods based on sentence embeddings like SetFit, though this is not explicitly mentioned in the original SetFit paper (Tunstall et al., 2022). SEPT improves both the Base model (R3 vs. R1) and the DAPT model (R4 vs. R2) by 7.3 and 6.2 points average, respectively.
By adding DAPT stage before SEPT, the classi- fication accuracy can be significantly increased by up to 6.7 points (on AMI) and 2.7 points average (R4 R3). However, we discussed in subsec- tion 3.3, executing the same SEPT procedure on every DAPT model results in computational ineffi- ciency. As a more efficient alternative, our AdaSent avoids repeating SEPT by sharing a SEPT adapter
6.3 Comparison of PEFT Methods
We experimented with four different PEFT meth- ods for both SEPT and DAPT (Figure 4). When applied to SEPT AdaSent, parallel adapter works best on the majority of the datasets (Ta-

---

Tuuable None
Adapter (4%)
Transtormer
All (100%)
Self-training
No
Yes
Parameters (0%)
(96%)
SEPT SEPT (AdaSent)
67.6
68.6 (+1.0)
Accuracy 65.0
69.0
71.2
71.5
71.5
72.4 (+0.9)
Table 5: Results of tuning subsets of model parameters (marked with relative sizes) in the final SetFit stage of AdaSent. None means only training the logistic regres- sion head.
Table 7: Averaged accuracy of AdaSent and SEPT, w/ or w/o self seffitrainin..
DAPT Steps
Cost (hour)
Method
Acc.
SEPT
DAPT
Total
MI.M
TSDAE
0.00
4.05
67.6
DAPT+SEPT (AdaSent)
69.7
67.3
100
0.44
4.49
69.8
DAPT~SSEPT
69.7
69.1
DAPT-SEPT
500
15 x 0.27
2.21
6.26
70.7
1000
4,42
8.47
71.1
Table 6: Averaged accuracy of different DAPT objec- tives in AdaSent and DAPT-
2000
8.83
12.88
71.5
0.00
0.17
67.9
0.44
0.61
69.5
AdaSent
500
x 0.17
2.21
2.38
70.7
1000
4.42
4.59
71.2
ble 13) and on average. Prefix-tuning is signifi- cantly worse than the other three methods. This might be due to the fact that the data in our SEPT dataset NLI+SC+SE come from three different tasks, whose properties cannot be compressed into single prefix. When applied DAPT in the SEPT SEPT-:DAPTPENT setting, their performance exhibits variability across different datasets (Ta- ble 13), but none the four PEFT methods this setting can beat the AdaSent variants due to the critical drawback of the setting discussed at the end of subsection 6.2.
2000
8.83
9.00
71.4
Table 8: Total training cost on 15 tasks with Distil- RoBERTa as base PLM on Tesla V100 GPU.
et al., 2015) to domain-specific data, as long as the same MLM objective is used. To verify this idea, we replace the MLM objective with TSDAE in both AdaSent and SAPT-SEPT. As shown in Table 6, using TSDAE instead of MLM in the DAPT stage of AdaSent leads a substantial de- crease of 2.4 points in the classification accuracy, while the performance drop in DAPT-sEPT is relatively marginal (0.6 on average). This supports our hypothesis that the adapter can only general- ize to collaborate with PLM parameters that are domain domain-adapted with the same objective as in the pre-training
6.4 Tunable Parameters in SetFit
We tune various subsets of parameters in the SetFit stage of AdaSent and compare the results in Ta- ble 5. We found that only updating the adapter parameters is not sufficient. However, tuning only the Transformer backbone leads to almost the same results as tuning all parameters (i.e. Transformer adapter). This indicates that with only few-shot labeled data, SetFit must at least update the Trans- former parameters to achieve good performance, and cannot work well on an adapter as in the case of SEPT, where much more supervised data are available.
6.6 Combining DAPT and Self-Training
Besides DAPT, another major way utilize the un- labeled data is self-training, which has been shown to be complementary to DAPT (Li et al., 2021b). To integrate self-training into SetFit, we first en- code the unlabeled data with the sentence encoder (in our case DAPT Transformer + SEPT adapter) trained with few-shot labeled data in the contrastive fine-tuning phase. When training the classification head, we iteratively pseudo-label the encoded un- labeled sentences and train with both the pseudo- labeled and the labella data
6.5 Explaining the Success of AdaSent
The success of AdaSent relies on the fact that a SEPT adapter trained on a base PLM can be un- problematically inserted into any domain-adapted version of the same PLM. This might be because in both original pre-training and Nomain-adaptivv pre-training, the PLM parameters are consistently tuned with the MLM objective. This implies that the adapter can generalize to work together with PLM parameters trained on different types of data, from languagl data (e.g. BookCorpus, Zhu
Table 7, we show that self-training can further improve both SEPT and AdaSent's accuracy by 1.0 and 0.9 on av- erage, respectively. These two close improvements reveal that the benefit of self- self-training is orthogonal to that of AdaSent/DAPT. We leave more complex
Whe training details are available in Appendix G.

---

approaches of combining AdaSent and self-training for future work.
cable for sentence pair classification like NLI. In addition, the advantage of SetFit is not significant in classification tasks with too many classes. More- over, as our method is based on sentence embed- dings, its application is limited to sentence classifi- cation, unlike other few-shot classification methods that can also handle token-level classification tasks like NER and POS tagging.
7 Training Cost
Table 8 gives an overview of the training cost for DAPT-SEPT and AdaSent in our experiments. We use Tesla V100 GPU for training. We leave out IMDB and LED as they have too long se- quences (ef. Table 2), thus cannot represent the majority of our tasks.
Another limitation is associated with the fact that the SEPT adapter in our method can only be in- serted into Nomain-adaptee language models with the same unmodified tokenizer and vocabulary as the original base PLM. For DAPT-ed models with a domain-specific tokenizer or vocabulary, we sup- pose the adapter trained on the original PLMM will not be compatible anymore.
With AdaSent, SEPT is trained once for 0.17h and the SEPT adapter can be shared across tasks. In contrast, SEPT costs 0.27 hours addi- tionally for every task due to its repeated SEPT. In experiments,
relatively small-sized data for SEPT. However, the SEPT cost increase dra- matically if much larger training data are used. For example, SEPT on the combination of all datasets in Table 10 for epoch can take 4 hours, resulting in 15 x 4 hours for DAPT-SSEPT for 15 tasks. For DAPT, we can see that 1000 steps are already sufficient for substantial improvement in accu- racy. In this case, AdaSent takes 4.59 hours for the training on 15 tasks in total, while DAPT-SEPT takes 8.47 hours (x 1.85).
Ethics Statement
Our experiments use publicly available datasets and benchmarks for training and evaluation, which are commonly used in the field of NLP. No personal information or sensitive data are involved in our work. Existing biases in the datasets pre-trained models can still be relevant concerns, since we do not specifically focus on mitigating them in the current work.
8 Conclusion
Acknowledgements
We introduce an efficient method to obtain domain- adapted sentence embeddings for few-shot clas- sification. We found that SttFii,, the previous state-of-the-art approach, can be significantly im- proved by introducing simple Domain-Adaptive Pre-Training (DAPT) stage before its Sentence- Embedding Fre-Training (SEPT). However, this DAPT approach requires the same SEPT procedure to be done on each DAPT-ed PLM for ev- ery domain, resulting in computational inefficiency. We propose a novel approach, AdaSent, to address this issue by storing the SEPT knowledge in an adapter that is trained on an unadapted PLM and in- sertable into any DAPT-ed PLM. AdaSent matches or surpasses the performance of DAPT->SEPT while significantly reducing the training cost of SEPT. We attribute the success of AdaSent to the gencralization ability of the SEPT udapter to work with PLM parameters trained on data from different domains with consistent MLM objective.
This work has been funded by HUAWEI Technolo- gies (Ireland) Co., Ltd. and by the German Federal Ministry of Education and Research (BMBF) under the promotional reference 13N15897 (MISRIK).
References
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. large anmo- tated corpus for learning natural language inference. In Proceedings of the 2015 Conference Empiri- cal Methods Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Compu- tational Linguistics.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems,
Limitations
Since our method is based on SetFit, it inherits some of its limitations. It is, for example, not appli-

---

Xiaochuang Han and Jacob Eisenstein. 2019. Unsu- pervised domain adaptation of contextualized em- beddings for sequence labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing Hong Kong. volume 33, pages 1877-1901. Curran Associates, Inc.
Computatiinnal Linguistics (Volume 1: Long Papers), pages 4277-4302 Toronto, Canada. Association for Computational Linguistics.
lñigo Casanueva, Tadas Temčinas. Daniela Gerz, Matthew Henderson, and Ivan Vulić. 2020. Efficient intent detection with dual sentence encoders. In Pro- ceedings of the 2nd Workshop on Natural Language Processing for Conversational AI, pages 38-45, On- line. Association for Computational Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021a. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meet- ing of the Association for Compulational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computa- tional Linguistics.
Mlas Chalkidis, Abhik Jana, Dirk Hartung, Michael Bommarito, Ion Androutsopoulos, Daniel Katz, and Nikolaos Aletras. 2022. LexGLUE: A benchmark dataset for legal language understanding in English. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pages 4310-4330 1030030,,000030 Dublin, Ireland. Association for Computational I Linguisties.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 20. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Process- ing, pages 6894-6910, 10, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Yiming Chen, Yan Zhang, Chen Zhang, Grandee Lee, Ran Cheng, and Haizhou Li. 2021. Revisiting self- training for few-shot learning of language model. In Proceedings of the 2021 Conference on Empir- ical Methods in Natural Language Procexsing, pages 9125- 9135, Online and Punta Cana, Dominican Re- public. Association for Computational Linguistics.
Harsha Gurulingappa, Abdul Mateen Rajput, Angus Roberts, Juliane Fluck, Martin Hofmann-Apitius, and Luca Toldo 2012. Development corpus to support the automatic extraction of drug- related adverse effects from medical case reports. Journal of Biomedical Informatics, 45(5):885 892. Text Mining and Natural Language Processing in Pbarmacogenomics.
benchmark
David Cox. 1958. The regression analysis of binary sequences. Journal of the Royal Statistical Society: Series B (Methodological), 20(2):215-232
Suchin Gururangan. Ana Marasović. Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342 8360, Online. Association for Computational Linguistics.
Franck Dernoncourt and Ji Young Lee. 2017. PubMed 200k RCT: a dataset for sequential sentence clas- sification in medical abstracts. In Proceedings of the Eighth International Joint Conference on Natu- ral Language Processing (Volume 2: Short Papers), pages 308-313, Taipei, Taiwan. Asian Federation of Natural Language Processing.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
(EMNLP-IJCNPP,, pages China. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022. Towards a unified view of parameter-efficient transfer learning. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 29, 2022. OpenReview.net.
Katja Filippova and Yasemin Altun. 2013. Overcom- ing the lack of parallel data in sentence compression. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1481-1491, Seattle, Washington, USA. Association for Computational Linguistics.
Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun hsuan Sung. Laszlo Lukacs, Ruiqi Guo, Sanjiv Ku- mar, Balint Miklos, and Ray Kurzweil. 2017. Effi- cient natural language response suggestion for smart reply. ArXiv, 1 006.0.
Jack FitzGerald, Christopher Hench, Charith Peris, Scott Mackie, Kay Rottmann, Ana Sanchez, Aaron Nash. Liam Urbach, Vishesh Kakarala, Richa Singh, Swetha Ranganath, Laurie Crist, Misha Britan, Wouter Leeuwis. Gokhan Tur, and Prem Natara- jan. 2023. MASSIVE: A IM-example multilin- gual natural language understanding dataset with 51 typologically-divee diverse languages. In Proceedings of the 61 Annual Meeting of the Association for
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In

---

Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Alexander Liu and Samuel Yang. 2022. Masked autoen- coders as the unified learners for pre-trained sentence representation. ArXiv, abs/2208.000 00231...........
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mo- hta, Tenghao Huang. Mohit Bansal, and Colin A Raf- fel. 2022. Few-shot parameter-efficien fine- tuning is better and cheaper than in-context learning. In Advances in Neural Information Processing Systems. volume 35, pages 1950-1965 Inc.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022. Virtual Event, April 25- 29, 2022. OpenReview.net.
Curran Associates,
Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzich Sacidi, Veselin Stoyanov, and Majid Yazdani. 2022. Prompt- free and efficient few-shot learning with language models. Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers). pages 3638-3652, Dublin, Ireland. Association for Computational Linguistics.
Yinhan Liu, Myle Out, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019, Roberta: robustly optimized BERT pretraining approach. CoRR, 19s//190.11669 16692.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Gregory Koch, Richard Zemel, Ruslan Salakhutdinov, al. 2015, Siamese neural networks for one-shot image recognition. In ICML deep learning workshop, volume 2. Lille.
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma. and Percy Liang. 2022. Fine- tuning can distort pretrained features and underper- form out-of-distributt Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29 29, 2022. OpenReview.net.
Pekka Malo, Ankur Sinha, Pekka J. Korhonen, Jyrki Wallenius, and Pyry Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. J. Assoc. Inf. Sci. Technol. 65(4):882-79)
In The Tenth International
Julian McAuley and Jure Leskovec. 2013. Hidden fac- tors and hidden topies: Understanding rating dimen- sions with review text. In Proceedings of the 7th ACM Conference on Recommender Systems, RecSys '13, page 165-172, New York, NY, USA. Association for Computing Machinery.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. 2019. BioBERT: a pre-traie pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):2234-
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. MTEB: Massive text embedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2014-2037, Dubrovnik, Croatia. Association for Computational Linguistics.
Haoran Li, Abhinav Arora, Shuohui Chen, Anchit Gupta, Sonal Gupta, and Yashar Mehdad. 202 la. MTOP: comprehensive multilingual task-oriented semantic parsing benchmark. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 2950-2962 Online. Association for Computa- tional Linguisties.
James O' O'Neil,, Polina Rozenshtein, Ryuichi Kiryo, Mo- toko Kubota, and Danushka Bollegala. 2021. I wish would have loved this one, but didn't multilin- gual dataset for counterfactual detection in product review. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pages 7092-7108, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Shiyang Li, Semih Yavuz, Wenhu Chen, and Xifeng Yan. 2021b. Task- adaptive pre- pre-training and self- trainini are complementary for natural language understand- ing. In Findings of the Association for Computa- tional Linguistics: EMNLP 2021, pages 1006-1015, Punta Cana, Dominican Republic, Association for Computational Linguistics.
Raj Nath Patel, Edward Burgin, Haytham Assem, and Sourav Dutta, 2021. Efficient multi-lingual sen- tence classification framework with sentence meta encoders. 2021 IEEE International Conference on Big Data (Big Data), pages 1889-1899.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Longuage Processing \Volume I: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021. True shw-shot learning with language models. In Ad- vances in Neural Information Processing Systems 34:

---

Annual Conference on Neural Information Process- ing Systems 2021, NewrlPS 2021. December 6-14, 2021, virtual, pages I 1054-11070.
Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021. Improving and simplifying pattern exploiting training. In Proceed- ings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4980-4991, Online and Punta Cana, Dominican Republic. Asso- ciation for Computational Linguisties.
Christian S. Perone, Roberto Silveira, and Thomas S. Paula. 2018. Evaluation of sentence embeddings in downstream and linguistic probing tasks. ArXiv, abs/1 806.06259.
Don Tuggener, Pius Däniken, Thomas Peetz, and Mark Cieliebak. 2020. LEDGAR: A large-scale multi-lahel corpus for text classification of legal pro- visions in contracts. In Proceedings of the Twelfth Language Resources and Evaluation Conference. pages 1235-1241, Marseille, France. European Lan- guage Resources Association.
Jonas Pfeiffer, Ivan Vulić, Iryna Gurevych, and Se- bastian Ruder. 2020. MAD-X: An Adapter- Based Framework for Multi-Task Cross- LongulL Transfer. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7654-7673, Online. Association for Computa- tional Linguistics.
Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo. Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022. Efficient few-shot learning without prompts. ArXiv, abs/2209.11055
Guangyuan Piao. 2021.
Scholarly text classification with sentence BERT and entity embeddings. In Trends and applications in knowledge discovery and data mining. pages 79-87, Cham. Springer Interna- tional Publishing.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3982-3992, Hong Kong, China. Association for Com- putational Linguistics.
Kexin Wang, Nils Reimers, and Iryna Gurevych. 2021. TSDAE: Using transformer-based sequential denois- ing auto-encoderfor unsupervised sentence embed- ding learning. In Findings of the Association for Computational Linguistics: EMNIP 2021, pages 671-688 Punta Cana, Dominican Republic. Associa- tion for Computational Linguistics.
Vin Sachidananda, Jason Kessler, and Yi-An Lai. 2021. Efficient domain adaptation of language models via adaptive tokenization. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pages 155-165, Virtual. Association for Computational Linguistics.
Kexin Wang. Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 2345-2360, Seattle, United States. Association for Computational Linguistics.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, abs/1910001
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A -overrae challenge corpus for sen- tence understanding through inference. In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguis- tics.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018 CARER: Con- textualized affect representations for emotion recog- nition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium. Association for Computational I Linguistics.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Asso- ciation for Computational Linguistics: Main Volume, pages 255-269, Online. Association for Computa- tional Linguistics.
Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thame Thang Lu- ong, and Quoc V. Le. 2020. Unsupervised data aug- mentation for consistency training. In Proceedings of the 34th International Conference on Neural In- formation Processing Systems, NIPS' 20, Red Hook, NY, USA. Curran Associates Inc.
Hongjin Su, Weijia Shi. Jungo Kasai, Yizhong Wang. Yushi Hu, Mari Ostendorf, Wen tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: iettructionfinetunnee text em- beddings. ArXiv, abs/22 2.09741.
Yunzhi Yao, Shaohan Huang, Wenhui Wang. Li Dong. and Furu Wei. 2021. Adapt-and-distill: Developing small, fast and effective pretrained language models for domains. In Findings of the Association for Com- putational Linguistics: ACL-IJCNLP 2021, pages

---

460-470, Online, Association for Computational Lin- guistics.
Method
Used Implementation
PEPT
https://github.c
adapter-trannformers
Jingtao 7han, Qingyao Ai, Yiqun Liu, Jiaxin Mao, Xiao- hui Xie, Min Zhang, and Shaoping Ma. 2022. Disen- tangled modeling domain and relevance for adapt- able dense retrieval. CoRR, (0./2208.05553.
TSDAE
https////gtthubb.cmm//kaLabbbb com/UKPLab/
sentence-transformers
SEPT
sentence-transfoormes
SimCSE
https://github.com
httpe//wb..oo//p..ettttooo-oooee Si SiMCSE
Lei Zhao and Cheng Yao. 2022. EICO: Improving few-shot text classification via explicit and implicit consistency regularization. In Findings of the Asso- ciation for Computational Linguisties; ACL 2022, pages 3582-3587, Dublin, Ireland. Association for Computational Linguistics.
MLM
https://githin
com/huugiigface//
no no_trainer...
SetFit
hoas:://iihuule htt.ithu....ohuugiiieeeee con/huggingface/ setfit
Yukun Zhu, Ryan Kiros, Richard S. Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015, Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In 2015 IEEE Interna- tional Conference on Computer Vision, ICCV 2015, Santiago, Chile, December 7-13, 2015, pages 19-27. IEEE Computer Society.
Table 9: Implementation used in this work.
(DAPT>>EPP and DAPT+SEPT ada) improves the performance of SEPT by around points on av- erage. Replacing full SEPT with SEPT adapter causes slight drop of around 0.5 in the classi- fication accuracy. Interestingly, without any pervised sentence embedding pre-training, DAPT itself can outperform SEPT on some datasets (AC, ADE, LED).
A Implementation
See Table 9 for the implementation of methods used in this work.
E PEFT results
B Experiment with SEPT Datasets
Results individual datasets when using differ- PEFT methods as discussed in subsection 6.3 in our AdaSent method (DAPT+SEPTpur)) and SEPT->DAPTPEIT are shown in Table 13.
We experiment with different SEPT datasets to check their transferability to downstream tasks Ta- ble 10. On average, AIINLI, SentenceComppession and SrackkechangeDupliiceeeeeessse three datasets. The similarity between the SEPT data and the downstream data seems to have influence on the performance. For example, QA- related data YYahooAnswerTTlleAnnwws Stack- exchangeDuplicaaeeuusioon and Answer Answer) are especially beneficial for the classification tasks involving user utterances in dia- logues (BANK, AMI, AMS, MI, MD). Given this observation, one might want to search for the opti- mal SEPT datasets depending on certain types of classification tasks. Our adapter-based method ables efficient SEPT, which helps to ease the data selection.
are the top
Evaluation Datasets
Table 14 provides examples from each evaluation dataset.
G Self-Training Setting
In the SetFFit phase, we contrastively fine-tune the sentence embedding model with the few-shot data as before (subsection 3.1), but replace the normal Logistic Regression fitting with self-training both labeled and unlabeled data. For this, the SalFrainnngclassii fier from "eak with 10 iterations and threshold of 0.9. At each iteration, the classifier predicts the label of the un- labeled data. The pseudo-labeled data with a confi- dence score higher than the threshold are used to augment the training data in the next iteration.
DAPT Objectives and Training Order
Results on individual datasets are listed in Table 11,
D Results on DistilBERT
We report the results on DistilBERT in Table 12. Similar to DistilRoBERTa, DAPT with MLM
11 See https: t. net/examples/trainigg /arapAE html for information of the datascts.
moduless/generated/kkkeas sttpseeedddeelaeeeeeeessmeeeeeee SelfTrainingClassiiiee html
semi semiisupervvied.

---

SEPT Data
AC
BANK
AMI
AMS
MI
MD
EMO
IMDB
TSE
TC
ARM
Avg.
AIINIJ
65.5
75.5
63.5
72.1
74.2
90.2
84.9
62.7
64.8
43.5
67.8
SenteneCompeession
74.0
74.9
61.5
72.9
74.9
90.6
83.8
60.3
58.7
42.2
67.9
SimpleWiki
65,4
74.3
59.4
71.1
70.8
89.3
45.3
83.5
59.7
63.8
41.9
65,9
Altlex
68.4
74.5
39.6
71.1
72.0
89.2
45.6
81.8
57.7
62.6
41.6
65.8
QuoraDuplicaresTripless
73.6
75.3
60.9
71.1
75.0
89.5
44.6
81.0
58.0
61.8
42.0
66.6
CocoCaptions
58.4
74.3
58.7
71.2
71.6
89.6
45.5
60.3
51.2
58.4
37.7
61.5
Flickr30kCuptions
58.0
74.1
59.5
71.2
73.8
89.4
45.1
60.0
52.3
66.4
36.8
62.4
Yahoo/Answers TitleQuestion
69.0
75.2
60.5
72.4
75.5
90.6
46.3
83.4
55.1
58.6
41.6
66.2
71.5
75.1
61.2
73.7
75.5
90.9
44.9
80.9
55.6
52.1
41.5
65.7
ShuskexchanncDuppiisaccuuuusostu
72.0
75.1
64.0
73.6
77.9
90.2
46.1
77.2
58.2
6006
43.1
67.1
YabrorAtwwernQQussiioonnnneee
67.3
75.0
61.2
73.4
75.4
90.8
45.3
81.5
52.3
62.7
41.2
66.0
Table 10: Results on MTEB tasks of SEPT model trained on different datasets. The best scores are marked in bold and second best with underline. We sample 100K instances from each SEPT dataset and train for epoch.
AC
FPB
TENT
ADE:
Avg.
TSZME
DAPT
74.1
77.6
64.4
76.0
77.4
93.0
46.3
78.4
53.7
49.6
44.8
51.1
55.7
61.6
62.5
5888
67.8
64.3
SEPT-DAPPT
79.5
77.5
66.8
76.4
79.9
93.2
46.1
79.5
50.6
47.6
47.3
58.4
53.2
63.0
60.0
57.6
68.3
65.0
DAPT-SEPT
72.9
77.3
67.9
76.3
79.8
93.4
52.0
85.6
63.6
62.1
48.9
66.6
63.9
66.2
65.3
65.4
68.1
69
DAPT
71.2
75.3
61.9
73.0
75.6
89.9
46.8
78.5
59.2
57.2
44.5
58.8
57.3
60.5
61.3
63.6
64.3
64.6
SEPT DAPT
66.2
75.4
63.3
74.0
78.8
89.8
44.4
66.9
59.1
69.9
44.7
64,9
61.7
60.2
58.3
68.2
65.5
65.4
DAPT-SEPT
62.5
66.1
MLM
DAPT
61.8
76.6
61.8
75.7
77.0
93.2
38.0
64.8
52.7
60.3
44.2
53.2
58.3
63.9
63.9
46.4
67.3
62.3
DDPT
73.2
76.4
65.0
75.6
79.0
92.8
49.1
79.3
58.3
51.3
48.8
55,9
61.5
65.9
64.7
59.2
67.9
66.
DAPT-SEPT
72.7
78.0
67.4
77.0
88.4
93.7
49.9
85.5
63.9
65.3
50.9
66.6
63.8
66.8
66.5
66.9
67.7
69.7
SEPT
70.2
75.5
64.5
73.6
77.4
90166
51.8
84.2
63.3
61.8
43.3
65.5
60.8
61.8
64.0
64.3
64.6
66.9
Base
65.9
75.2
60.6
71.0
73.9
89.4
40.3
68.0
50.9
55.6
37.6
48.5
50.8
57.8
60.9
49,2
64.2
60.0
Table 11:
Comparison of different DAPT objectives and training order of DAPT and SEFT. The best scores are marked in bold and second best with underline. Note that the training settings of DAPT here is different from that in Table and Table 4: We do DAPT for 3 epochs instead of numbbrrmeer of steps.
Model
AC
BANK
AMI
AMS
MI
MD
EMO
IMDB TSE
TC
ARM
FPB
TENS
TENT
ADE
RCT
LED
Avg.
SEPT
74.1
75.6
37.9
49,6
53.2
DAPT
82.1
79.6
70.2
79.2
82.9
95.1
40.1
78.3
56.1
50.6
45,2
53.8
53.2
65.6
72.4
65.3
67.3
66.9
SEPT
SEPT
72.6
75.7
67.5
74.3
79.7
91,9
48.4
80.7
63.8
63.9
42.8
61.1
62.1
58.2
63.9
58.6
65.6
66.5
DAPT-SEPT
75.6
79.7
73,9
80.4
83.9
94.99
50.9
83.4
63.8
64.4
46.1
62.2
64.6
66.3
65.9
64.0
67.3
69.8
SEPT on Aduupter
SBPT
75.5
75.2
66.3
73.7
78.5
91.4
50.D
78.9
63.5
58.2
42,1
63.4
61.8
55.9
62.3
58.2
65
65.9
AdaSent
80.6
79.8
72.1
80.3
82.4
95.7
$1.6
82.2
62.8
56.6
47.1
60.4
66.2
63.4
64.9
65.8
67.3
69.4
Table 12: Results on DistiBBRTT. Best scores are in bold.
PEFT method
AC
BANK
AMI
AMS
MI
MD
EMO
IMDB
INE
TC
ARM
FPB
TENS
TENT
ADE
RCT
LED
Avg.
DAPT-SEPTrun
(AdaSent)
Bottieneck adupter
76.3
80.7
73.0
80.2
82.2
95.3
$1.5
87.4
64,2
58.6
48.0
65.0
66.7
68.6
65.2
68.7
68.2
70.6
Parallel adagter
77.9
80.5
67.5
LoRA
79.0
80.7
72.9
79.6
82.0
94.6
$2.9
85.1
63.1
60.1
46.3
63.5
64.7
66.9
67.1
65.7
67.55
70.1
Prefix-tuning
78.5
94.1
62.9
49.0
360
41.8
47.9
58.8
47.6
67.3
600
SEPT DEPTT-DAPPrrr
Bottleneck Adapter
76.4
77.5
67.0
74.6
79.6
91.5
50.3
82.9
61.4
58.0
43.5
60.7
63.7
63.6
64.6
65.1
65.8
67.4
Parallel Adapter
72.8
79.5
69.8
78.0
81.0
93.7
48.3
84.4
59.2
59.4
44.9
62.7
62.0
65.2
66.7
64.0
66.8
68.1
LoRA
27.5
77.3
66.7
73.6
78.5
91.2
51.0
83.8
63.2
58.9
43.6
61.1
60.3
63.6
63.3
64.2
65.5
67.3
Prefix.cuning
78.9
77.3
66.0
72.6
78.6
90.5
$2.9
84.7
62.6
63.7
44.2
64.5
67.7
62.7
63.8
63.0
65.8
68.2
Table 13: Results on individual datasets of different PEFT methods for DAPT+SEPTpEr (AdaSent) and
SEPT- Best scores are in bold for both models.

---

Dataset
Abbr
Text
Label
classification
Amazon
Counterfactual Neill el al., 2021)
AC
In person it looks as though it would have cost lot more.
counterfactual
Banking77 77
(Casanueva et al., 2020)
BANK
I am still waiting on my card?
card arrival
Amazon
Massive Intent
AMI
wake up at nine friday
alarm_sct
(FitzGerald et al., 2023)
Amazon
Massive Scenario
AMS
wake up nine friday
alarm
(FitzGerald et al., 2023)
MTOP Intent
et al., 2021a)
Has Angelika Kratzer video messaged me?
MESSAGE
MTOP Domain
(Lii et al., 2021a)
MD
Has Angelika Kratzer video messaged me?
messaging
Emotion
(Saravia et al., 2018)
EMO
ive been feeling little burdened lately wasnt sure why sadness that was
Imdb
(Maas al., 201 I)
IMDB
I rented I AM CURIOUS- YELLLO from my video store because of all the controversy that surrounded when first released in 1967. also heard that at first was seized by U.S. customs if it ever tried to enter this country, therefore being fan of films considered "controversial" really had to see this for myself <br /><hr />The plot is centered around young Swedish drama student named Lena who wants learn everything she about life. [..] AM CURIOUS YELLOW good film for anyone wanting study the and potatoes (no pun intended) Swedish einema. But really, this film doesn't have much of plot.
negative
Twitter
Sentiment
TSE
I'd have responded, were going
neutral
Extraction
Toxic
TC
theres enough going on around here for air service toxic want waste there time this town
Conversation
Amazon
Reviews Multi
ARM
received my first order this product and was broke so I ordered it again. The second one was broke more places than first. can't 't blame the shipping process as it's shrink wrapped and boxed.
(McAuley and Leskovec, 2013)
Domain specific tasks
Financial
With new production plant the company would in- Positive crease its capacity to meet the expected increase in de- mand would improve use of raw materials and therefore increase the production profitahility.
PhraseBank
Twitter
Financial
TFNS
Grubhub gains bear margin view
Bearish
News
Sentiment
Twitter
Financial
TENT
Analysts reveal the top stocks with 'significant upside potential" heading into earnings http://t.coffallKKawe K3nwAZ
Analyst Update
News Topic
Adverse
Drug
ADE
Intravenous writhrommyin-indduee induced ototoxicity.
Related
Events
PubMed
RCT
Outcome measures included pain reduction and improve- ment in function scores and systemic inflammation mark-
Methods
RCT
LEDGAR
LED
Except otherwise set forth this Debenture, the Com- pany. for itself and its legal representatives, and assigns, expressly waives presentment, protest, de- mand, notice of dishonor, notice nonpayment, notice of maturity, notice of protest, presentment for purpose of accelerating maturity, and diligence in collection.
Waivers
Table 14: Examples from evaluation datasets.


Reviews:
[{"review_id": "GKvaaQBSsH", "reviewer": "Reviewer_cXZX", "summary": "", "questions": "The author could address my concerns in the above section.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for recognizing that we have conducted comprehensive experiments to support our method for combining domain-adaptive pre-training (DAPT) and sentence-embedding pretraining (SEPT). We provide responses to reviewer’s concerns below. \n\n>   The novelty is limited. This paper is basically based on SetFit and PEFT, I observe limited technical contribution.\n\nThe proposed approach is indeed based on parameter-efficient training and SetFit. That alone, we argue, does not mean that what we propose is not novel. This work addresses the following important practical question for which no viable solution has yet been proposed: how to perform few-shot sentence classification in special domains effectively and efficiently? As we argue in the abstract, introduction, and Section 3.3, this cannot be achieved by trivially coupling domain adapters and SetFit (see, e.g., L55-63). \n\nConcretely, applying standard LM-based domain adaptation on top of a pretrained sentence encoder disrupts the model’s ability to semantically accurately embed sentences and leads to poor few-shot classification performance (i.e., not effective); Sentence-specialization of a previously domain-specialized regular encoder, on the other hand, would likely be effective but is not efficient as training sentence encoders would require millions of training instances for each domain.       \n\nOur modular solution, (1) stores the sentence-specialization abilities – obtained via single sentence-encoder training procedure in the general domain – into an adapter (efficient, single parameter-efficient sentence specialization training, regardless of the number of domains) and (2) combines that sentence-encoding adapter with domain-specialized LMs from various domains in modular fashion (which, as our experiments show, is effective, yielding SotA few-shot classification performance in various domains). Because of this, we believe that we are providing a novel solution to an important practical problem, even if the building blocks of that solution are known.    \n\n> In an era of Large Language Models, studying domain-related training approach of encoder-only models appears somewhat limited. The author could potentially extend their methodology to encoder-decoder or decoder-only models.\n\nWhile one could, naturally, perform few-shot sentence classification with generative LLMs, either via in-context learning or PEFT, this would not be computationally justified in most practical setups. Most sentence classification problems in practice (e.g., on mobile or edge devices) are not solved via fine-tuning of even the smaller, traditional PLMs (BERT & co.): this is in part because few-shot setups do not offer enough data for full fine-tuning and, more importantly, due to the limited compute available. This is why sentence encoders are primarily used as feature providers and only coupled with a linear classifier. LLMs are amazing generalizers (to new tasks), but one would in practice not use GPT-4 or Llama-v2 to classify, e.g., billions of tweets into many product-related categories due to an immense inference cost in terms of both time and money.  \n\nAnother advantage of encoder-only models that produce “ready” text/sentence embeddings in real-world use case is that the features they produce can easily be combined with other (non-textual) features that are helpful for classification, e.g. meta information. This is why, even in the era of LLMs, encoder-only models will continue to play a significant role in practice.    \n\nWe agree with the reviewer that domain adaptation of encoder-decoder or decoder-only models for text generation tasks is an important topic, but  it is out of scope of this paper. Our focus in this work is on a discriminative task of few-shot sentence classification.\n\n> In Section 3.2, the supervised contrastive learning for pretraining is not so intuitive. Firstly, the definition of similarity between sentences should not solely rely on having the same task label, the author should also consider the semantics of two sentences. Secondly, if the author employs the supervised contrastive learning, instances with identical labels will be pushed closer, this might cause the representation to collapse into a confined space, thereby hurting the generalization ability. Thirdly, simply aligning two sentences with the same label in the representation space would introduce noise and confusion for pretrained language models, especially considering that sentences with the same label can have different semantics. Additionally, even when the two sentences have similar semantics, we still cannot align them due to the sensitivity of pretrained encoder. A single change in punctuation would bring label flipping in pretrained encoder in some cases, let alone the combinations of tokens from two sentences are completely different. Therefore, pushing two sentences closer may confuse the pretrained encoder.\n\nWe want to clarify some misunderstandings here. Both sentence embedding pre-training (SEPT, Section 3.2) and downstream classification task fine-tuning (SetFit, Section 3.1) use contrastive learning methods, but the two methods are not the same because they serve different goals. We will clarify this further in the text. \n\nThe idea of Supervised Contrastive Learning (SCL) that the reviewer is primarily referring, and which is the first step of the SetFit approach (not our contribution), is an established method for classification tasks in low-data setups: the aim is to push instances of the same classes closer together in the representation space and those from different classes further apart (in other words, to reshape the representation space of the encoder so to cluster instances according to their class labels). Note that sentences of different classes are pushed away from each other in the meantime, so there won’t be a collapse in the representation space. Because the goal of training a classifier is indeed to better distinguish between classes, SCL is used to create better separated clusters in the representation space, which in turn means a clearer decision boundary for the classifier. This will, indeed, reduce the generalizability of the encoder, but here we do not aim to obtain a general-purpose encoder, but one exactly specialized for this particular sentence classification task.  \n\nHowever, the contrastive learning method described in Section 3.2 is different from SCL. The purpose of sentence-embedding pre-training (SEPT) is to train universal semantic representations that can be fine-tuned in different downstream tasks. Here, we pull sentences with similar meaning closer together (and not sentences with the same task label as in SCL). Data for contrastive training of sentence encoders is typically obtained from NLI datasets (sentence pairs with the “entailment” label) or paraphrase datasets. This has been the backbone of all prominent sentence embedding models, e.g., SBERT (Reimers & Gurevych, 2019), SimCSE (Gao et al., 2021).  \n\nAll that being said, the contribution of our work does not lie in SCL/SetFit (Section 3.1) nor SEPT (Section 3.2). We are using these methods without modification, because they have been proven very successful in previous work. As acknowledged by the reviewer, the main contribution of our work to combine domain-adaptive pre-training (DAPT) and sentence embedding pre-training (SEPT) in a modular fashion so that the procedures are disentangled and applicable efficiently to a wide range of domains (i.e., so that the heavy SEPT has only to be done once for all domains, using the data from the general domain). \n\n> References\n\nNils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Computational Linguistics.\n\nTianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.SimCSE: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6894–6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Comprehensive experiments of different methods for combining domain-adaptive pretraining and sentence embedding pretraining.", "reasons_to_reject": "1. The novelty is limited. This paper is basically based on SetFit and PEFT, I observe limited technical contribution.\n2. In an era of Large Language Models, studying domain-related training approach of encoder-only models appears somewhat limited. The author could potentially extend their methodology to encoder-decoder or decoder-only models.\n3. In Section 3.2, the supervised contrastive learning for pretraining is not so intuitive. Firstly, the definition of similarity between sentences should not solely rely on having the same task label, the author should also consider the semantics of two sentences. Secondly, if the author employs the supervised contrastive learning, instances with identical labels will be pushed closer, this might cause the representation to collapse into a confined space, thereby hurting the generalization ability. Thirdly, simply aligning two sentences with the same label in the representation space would introduce noise and confusion for pretrained language models, especially considering that sentences with the same label can have different semantics. Additionally, even when the two sentences have similar semantics, we still cannot align them due to the sensitivity of pretrained encoder. A single change in punctuation would bring label flipping in pretrained encoder in some cases, let alone the combinations of tokens from two sentences are completely different. Therefore, pushing two sentences closer may confuse the pretrained encoder.", "questions_to_authors": "The author could address my concerns in the above section.", "paper_topic_and_main_contributions": "This paper proposes applying an adapter module of sentence embedding pretraining to avoid the overhead of repeating computing of domain adaptive pretraining. The main contribution lies in that the author proposes a training approach combining the domain adaptive pretraining and sentence embedding pretraining.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TGze00irIx", "reviewer": "Reviewer_gtNi", "summary": "", "questions": "Please address the concerns raised in \"Reasons to Reject\"", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased by the reviewer’s positive comments on the novelty, versatility and effectiveness of our proposed approach. We appreciate that the reviewer recognizes the importance of domain adaptation and few-shot learning in low-resource scenarios.\n\nWe want to point out a few misunderstandings. First, our approach is not for “NLI tasks” and we never mention our text embeddings are “instruction-finetuned” (in accept reason 1). To clarify, our end task is sentence classification, not “sequence labeling” (in accept reason 2).\n\nBelow we address the individual concerns raised by the reviewer. \n\n> Lack of novelty: The paper does not introduce any significant novel contributions to the field. The proposed approach for natural language inference (NLI) tasks and the unsupervised domain adaptation technique for contextualized embeddings have been previously explored in the literature. The paper fails to demonstrate how its approach differs from existing methods and why it is superior.\n\nWe want to clarify a misunderstanding here: Our task is not NLI (which is a sentence pair classification task), but single sentence classification based on sentence embeddings. We only use NLI data as the training data – similar/dissimilar sentence pairs – in order to obtain a sentence encoder, not to solve the NLI task itself.\n\nRegarding the concern of the lack of novelty, please see also the reply to Reviewer 1. The research gap we address in our paper is the lack of an efficient way to combine sentence embedding pre-training (SEPT) and domain-adaptive pre-training (DAPT) in order to enable effective (i.e., well-performing) few-shot sentence classification in various domains. \n\nAlthough these two techniques have been explored separately in the literature for other tasks and models, there is no easy way to combine both, especially for the SOTA few-shot classification approach SetFit (not our contribution). Simply doing domain adaptation and sentence embedding training sequentially has severe efficiency issues. We explained the limitations of existing solutions in the Introduction (line 55-63), and in more details, in Section 3.3. We illustrated the differences between our AdaSent and other methods in Figure 2, and explained them in Section 4.2. Our approach is superior, because we eliminate the need for repeating the same sentence embedding pre-training for every new domain by disentangling SEPT from DAPT, while obtaining strong performance. These efficiency gains of AdaSent is more pronounced the more domains there are for which need effective few-shot sentence classification (See Section 7 for comparison of computational cost of our AdaSent vs. the alternatives). Because of this, we believe that we are providing a novel solution to an important practical problem (few-shot sentence classification in specialized domains under computational constraints), even if the individual building blocks of that solution are known.  \n\n> Insufficient experimental evaluation: The paper lacks a comprehensive and rigorous experimental evaluation. The evaluation section does not provide detailed comparisons with state-of-the-art methods or baselines, making it difficult to assess the effectiveness and performance of the proposed approach. Additionally, the paper does not provide statistical significance tests or ablation studies to support the claims made.\n\nWe provided 3 baselines (Section 5, line 274-280), which are the vanilla PLM, a domain-adapted PLM without sentence embedding pre-training, and a pre-trained sentence transformer without domain adaptation. The comparison of these baselines with our methods (Table 3 and 4) serves as an ablation study, proving the effectiveness of the individual components of our method: domain-adaptive pre-training (DAPT) and sentence-embedding pre-training (SEPT). In addition, we did controlled experiments on 3 domain adaptive objectives (Section 6.1) and 4 parameter-efficient fine-tuning (PEFT) methods (Section 6.3), showing the MLM objective for domain adaptation and the parallel adapter module for SEPT is better than other alternatives. In our main experiments, we proved the effectiveness of our method that meaningfully combines DAPT and SEPT through the comparison with 3 other combination strategies (Tables 3 and 4).\n\nOur work is built upon the previous SOTA, SetFit (Row 3 in Table 3/4), which is also the most important baseline in our experiments. In that sense, we think these results are sufficient to provide meaningful information about acknowledging/understanding the advantage of our approach. In its original work, SetFit is compared with other approaches, e.g. PERFECT, ADAPET, T-FEW, etc. and is shown to have superior performance over these counterparts. In our work, the results show that our AdaSent is better than SetFit while conserving its efficiency. For other necessary approaches, it would be great if you could name them. \n\nStatistical significance: Thanks for the suggestion. We have now carried out the statistical testing of the performance differences between the models and the revised version will include significance results in Section 6.2. We also put the findings here. First, we did t-tests with significance level 0.05 to support the claim that replacing full SEPT with adapter-based SEPT has no negative impact on the performance, while being much more efficient. On all datasets in our evaluation experiments, full SEPT and adapter-based SEPT (R3 vs. R5 in Table 3 and 4) achieve comparable accuracies with no significant difference. This is also true in the case of domain-adapted models (R4 vs. R7 in Table 3 and 4), except for one dataset AMS, where Adasent is even significantly better than DAPT→SEPT. Secondly, t-tests on SEPT baseline (which is used in original SetFit) and our AdaSent (R3 vs. R7 in Table 3 and 4) demonstrate that AdaSent statistically significantly (t-tests with significance level 0.05)  improves the performance on 8 of the evaluation datasets. \n\n> Inadequate clarity and organization: The paper suffers from poor writing quality, making it challenging to understand the proposed approach and replicate the experiments. The organization of the paper is unclear, with important details and explanations missing in several sections. The lack of clarity hinders the reproducibility and understanding of the work.\n\nWe thank the reviewer for the comment, and would improve the paper organization and details for better readability. We would greatly appreciate it if the reviewer could be more specific about the parts and aspects of the writing that seem particularly confusing and poorly organized.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a novel approach for natural language inference (NLI) tasks. It introduces a method for instruction-finetuned text embeddings that can be applied to various tasks, demonstrating the versatility and effectiveness of the proposed approach.\n\n2. The paper addresses the important problem of domain adaptation in natural language processing (NLP). It presents an unsupervised domain adaptation technique for contextualized embeddings, which can improve the performance of sequence labeling tasks in different domains.\n\n3. The paper contributes to the field of few-shot learning with language models. It introduces a prompt-free and efficient few-shot learning method using language models, which can be valuable for scenarios where limited labeled data is available.", "reasons_to_reject": "1. Lack of novelty: The paper does not introduce any significant novel contributions to the field. The proposed approach for natural language inference (NLI) tasks and the unsupervised domain adaptation technique for contextualized embeddings have been previously explored in the literature. The paper fails to demonstrate how its approach differs from existing methods and why it is superior.\n\n2. Insufficient experimental evaluation: The paper lacks a comprehensive and rigorous experimental evaluation. The evaluation section does not provide detailed comparisons with state-of-the-art methods or baselines, making it difficult to assess the effectiveness and performance of the proposed approach. Additionally, the paper does not provide statistical significance tests or ablation studies to support the claims made.\n\n3. Inadequate clarity and organization: The paper suffers from poor writing quality, making it challenging to understand the proposed approach and replicate the experiments. The organization of the paper is unclear, with important details and explanations missing in several sections. The lack of clarity hinders the reproducibility and understanding of the work.", "questions_to_authors": "Please address the concerns raised in \"Reasons to Reject\"", "paper_topic_and_main_contributions": "This paper introduces AdaSent, a method for domain-adapted sentence embeddings for few-shot classification. It combines Domain-Adaptive Pre-training (DAPT) and Sentence Embedding Pre-training (SEPT) to improve classification accuracy. AdaSent achieves comparable or better performance than full SEPT on DAPT-ed PLMs while reducing training costs. Experimental results on 17 different few-shot classification datasets demonstrate the effectiveness of AdaSent.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "4950p193Ss", "reviewer": "Reviewer_jeTA", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the precise understanding of our work and the insightful feedback. We are encouraged that the reviewer recognizes the key contribution of our work: we proposed an efficient approach to train a sentence embedding adapter which is reusable on top of any domain-adapted LM, avoiding to incur a large computational cost of sentence specialization of the LM for each domain. We appreciate the reviewer’s positive comments on the strong evidence provided by our extensive experiments, high writing quality, and sufficient reasoning for the improved performance. We are grateful for the reviewer’s suggestion on improving our writing and will definitely correct the typos in our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Important contribution: The ability to train the sentence embedding adapter only once and use it with the other domain-adapted LMs is significant. Performing domain-specific adaptation is also a good contribution but not a surprising result.\n\nExtensive experiments: The authors performed experiments across many datasets, giving strong evidence for their approach.\n\nClear writing: The paper is clear and organized well. The authors explain their reasoning for improved performance of their approach over full finetuning (with additional experiments), which I thought was necessary for the paper since it is a somewhat counterintuitive finding.\n\nCitation of previous work: The authors do a great job citing previous work to explain things throughout their paper.", "reasons_to_reject": "Seems like a solid paper. I can't think of any real reason to reject other than small nitpicky things.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper discusses an adapter technique that avoids the need to finetune on the sentence embedding objective for each domain-adapted language model. The proposed approach is efficient in that the sentence embedding objective only needs to be trained once but can be used with all domain-adapted models. The approach also leads to slight improvements over full finetuning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "You should write \"Parameter Efficient Fine-Tuning (PEFT)\" for your first usage of PEFT. I think you wrote the full acronym for everything else but just make sure because some people may not know what these things stand for.\n\nLine 408 \"proofs\" --> \"proves\"\n\nLine 458 \"Transfomrer\" --> \"Transformer\"\n\nI think there were a few more small typos, just proofread it again before the camera-ready submission."}}]

Author rebuttal:

Meta-review: There is a consensus among the reviewers that the paper's contribution to the field of few-shot learning is valuable. They also agree on the experiments being comprehensive and thorough. Two of the reviewers initially had concerns with regard to the clarity of the paper and the lack of statistical significance tests. Upon discussion, their concerns have partially been addressed. The authors clarified some points on contrastive learning and on how this work is positioned with respect to existing techniques in terms of novelty. Also, the authors are going to provide new results on the significance tests, and have promised to work hard on improving the writing and structure of the paper (which has been an issue raised in two of the reviews).
Decision: Accept-Main