Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models
Authors: Alejo Lopez-Avila
Víctor Suárez-Paniagua
Venue: EMNLP 2023 Main (2023)
Keywords: transfer learning
contrastive learning
denoising autoencoders
NLU
TL;DR: New technique for transfer learning combining contrastive learning with autoencoders and data augmentation

Abstract: Recently, using large pre-trained Transformer models for transfer learning tasks has evolved to the point where they have become one of the
flagship trends in the Natural Language Processing (NLP) community, giving rise to various outlooks such as prompt-based, adapters, or combinations with unsupervised approaches, among many others. In this work, we propose a 3-Phase technique to adjust a base model for a classification task. First, we adapt the model’s signal to the data distribution by performing further training with a Denoising Autoencoder (DAE). Second, we adjust the representation space of the output to the corresponding classes by clustering through a Contrastive Learning (CL) method. In addition, we introduce a new data augmentation approach for Supervised Contrastive Learning to correct the unbalanced datasets. Third, we apply fine-tuning to delimit the predefined categories. These different phases provide relevant and complementary knowledge to the model to learn the final task. We supply extensive experimental results on several datasets to demonstrate these claims. Moreover, we include an ablation study and compare the proposed method against other ways of combining these techniques.

Combining Denoising Autoencoders with Contrastive Learning to fine-tune Transformer Models
Alejo López-Avila
Victor Suárez-Paniagua Huawei Ireland Research Center
Huawei London Research Centre London, UK
Dublin, Ireland
victor. suarer.suaez..ee. www.ree.paaaagaeeuaaaaaaaaaaaaaa wartterrs.aptttee.te.ammm
Abstract
reasonable choice. This paradigm has raised in- terest in improving these techniques, either from the point of the architecture (Qin et al., 2019), by altering the definition of the fine-tuning task (Wang et al., 2021b) or the input itself (Brown et al., 2020), but also by revisiting and proposing different self- supervised training practices (Gao et al., 2021). We decided to explore the latter and offer novel ap- proach that could be utilised broadly for NLP clas- sification tasks. We combine some self self-supervised methods with fine-tuning to prepare the base model for the data and the task. Thus, we will have a model adjusted to the data even before we start fine-tuning without needing to train model from scratch, thus producing better results, as shown in the experiments.
Recently, using large pre-tree vre-trained Transformer models for transfer learning tasks has evolved the point where they have become of the flagship trends in the Natural Language Pro- cessing (NLP) community, giving rise to vari- outlooks such as prompt-based, adapters, or combinations with unsupervised approaches, among many others. In this work, we propose a 3- Phase technique to adjust base model for a classification task. First, we adapt the model's signal the data distribution by per- forming further training with a Denoising Au- toencoder (DAE). Second, we adjust the repre- sentation space of the output the correspond- ing classes by clustering through a Contrastive Learning (CL) method. addition, we intro- duce new data augmentation approach for Su- pervised Contrastive Learning to correct the un- balanced datasets. Third, we apply fine-tuning to delimit the predefined categories. These dif- ferent phases provide relevant and complemen- tary knowledge the model learn the final task. We supply extensive experimental results on several datasets to demonstrate these claims. Moreover, we include ablation study and compare the proposed method against other ways of combining these techniques.
A typical way to adapt neural network to the input distribution is based on Autoencoders. These systems introduce a bottleneck for the input data distribution through reduced layer, a sparse layer activation, or a restrictive loss, forcing the model to reduce a sample's 's representation at the narrowing. A more robust variant of this architecture is DAE, which corrupts the input data to prevent it from learning the identity function. The first phase of the proposed method is DAE (Fig. la), replacing the final layer of the encoder with one more adapted to the data distribution.
1 Introduction
The never-ending starvation of pre-trained Trans- former models has led to fine-tuning these models becoming the most common way to solve target tasks. A standard methodology uses a pre-trained model with self-supervised learning on large data base model. Then, replace/increase the deep layers to learn the task in supervised way by leveraging knowledge from the initial base model. Even though specialised Transformers, such Pe- gasus (Zhang al., 2020) for summarisation, have appeared in recent years, the complexity and re- sources needed to train Transformer models of these scales make fine-tuning methods the most
Contrastive Learning has attracted the attention of the NLP community recent years. This family of approaches is based on comparing an anchor sample to negative and positive samples. There are several losses in CL. like the triplet loss (Schroff et al., 2015), the contrastive loss (Chopra et al., 2005), or the cosine similarity loss. The second phase proposed in this work consists of Con- trastive Learning using the cosine similarity (as shown in Fig. lb). Since the data labelled, we use supervised approach similar to the one presented in (Khosla et al., 2020) but through Siamese Neu- ral Networks. In contrast, we consider Contrastive

---

Learning and fine-tuning the classifier (FT) as two distinct stages. We also add a new imbalance cor- rection during data augmentation that avoids over- fitting. This CL stage has a clustering impact since the vector representation belonging to the same class will tend to get closer during the training. We chose some benchmark datasets in classification tasks to support our claims. For the hierarchical ones, we can adjust labels based the number of similar levels among samples.
Autoencoders were introduced in (Kramer, 1991) and have been a standard for self-supervised learning Neural Networks since then. However, new modifications were created with the explo- sion of Deep Learning architectures, such as DAE (Vincent et al., 2010) and masked Autoencoders (Germain et al., 2015). The Variational Autoen- coder (VAE) has been applied for NLP in (Miao et al., 2015) (Li al., 2019) with RNN net- works DAE with Transformers (Wang al., 2021a). Transformers-
Finally, once we have adapted the model to the data distribution in the first phase and clustered the representations in the second, we apply fine-tuning at the very last. Among the different variants from fine-tuning, we use the traditional one for Natu- ral language understanding (NLU), i.e., add small Feedforward Neural Network (FNN) as the classifier on top of the encoder with two layers. We use only the target task data without any auxiliary dataset, making our outlook conf-contai The source code is publicly available at GitHub!. To summarise, our contribution is fourfold:
Sequential Denois- ing Auto-Encoder (Wang et al., 2021a) is an unsu- pervised method for encoding the sentence into a vector representation with limited or no labelled data, creating noise through an ML. task. The authors of that work evaluated their approach on the following three tasks: Information Retrieval, Re-Ranking, and Paraphrase Identification, show- ing an increase of up 6.4 points compared with previous state-of-the-art approaches. (Savinov et al., 1021), the authors employed a new Trans- former architecture called Step-unrolled Denoising Autoencoders. In the present work, we will apply DAE approach to some Transformer models and extend its application to mateece--bssed and more general classification tasks.
1.
We propose a 3-Phase fine-tuning approach to adapt pre-trained base model to su- pervised classification task, yielding more favourable results than classical fine-tuning.
The first work published Contrastive Learn- ing (Chopra al., 2005). After that, several versions have been created, like the triplet net in Facenet (Schroff et al., 2015) for Computer Vi- sion. The triplet-loss compares given sample and randomly selected negative and positive samples making the distances larger and shorter, respec- tively. One alternative approach for creating pos- itive pairs is slightly altering the original sample. This method was followed by improved losses such as N N-pair Loss (Sohn, 2016) and the Noise Con- trastive Estimation (NCF) (Gutmann and Hyväri- nen, 2010), extending the family of CL. techniques. In recent years, further research has been done on applying these losses, e.g. by supervised methods such as (Khosla et al., 2020). which is the one that most closely resembles one in our second phase.
2. We propose an imbalance correction method by sampling noised examples during the aug- mentation, which supports the Contrastive Learning approach and produces better vector representations.
3. We analyze possible ways of applying the de- scribed phases, including ablation and joint loss studies.
4. We perform experiments several well- known datasets with different classification tasks to prove the effectiveness of our pro- posed methodology.
2 Related Work
Sentence-BERR (Reimers and Gurevych, 2019) employs Siamese Neural Network using BERT with pooling layer to encode two sentences into sentence embedding and measure their similarity score. Sentence-BERT was evaluated on Seman- tic Textual Similarity (STS) tasks and the SentE- val toolkit (Conncau and Kicla, 2018), outperform- ing other embedding strategies in most tasks.
One of the first implementations was presented (Reimers and Gurevych, 2019), an application of Siamese Neural Networks using BERT (Devlin et al., 2018) to learn the similarity between sen- tences.
'https: //gi http://ggtubb.to..tee. www.w//wwwwapan.aaa.o..oooo papaaa.. finetuning
2

---

DAE
DAECL
Recovered input
Cosine Similanty: sim(u.v)
Cross- Ennts-y Loss
DECODER
Classifier
POOLING
POOLING
DAECL
POOLING
DAECL
ENCODER
NOISE
LABELLED
LABELLED
DATA
PAIRS OF LABELLED DATA
(b)
(c)
Figure 1: (a) First stage: Denoising Autoencoder architecture where the encoder and the decoder are based on the pre-traine pre-trained Transformers. The middle layer following the pooling together with the encoder model will be the resulting model of this stage. (b) Second stage: Supervised Contrastive Learning phase. We add pooling layer to the previous one to learn the new clustered representation. The set of blocks denoted as DAECL will be employed and adapted base model in the fine-tuning phase. (c) Third stage: Classification phase through fine-tuning
our particular case, we also use Siamese Networks within the CL options. Similar to this approach, the Simple Contrastive Sentence Embedding (Gao et al., 2021) is used to produce better embedding representations. This unlabelled data outlook uses two different representations from the same ple, simply adding the noise through the standard dropout. addition, they tested it using entailment sentences as positive examples and contradiction sentences as negative examples and obtained better results than SBERT in the STS tasks.
using mostly unlabelled data despite some little labelled data. In many state-of-the-ar procedures, like (Sun et al., 2020a), other datasets than the tar- get dataset are used during training. In our case, only the target dataset.
3 Model
In this section, we describe the different proposed phases: Denoising Autoencoder that makes the inputs robust against noise, Contrastive Learning approach identify the similarities between differ- ent samples in the same class and the dissimilarities with the other class examples together with novel imbalance correction, and finally, the traditional fine-tuning of the classification model. In the last part of the section, we describe another approach combining the first two phases into one loss.
Whereas until a few years ago, models were trained for a target task, the emergence of pre- trained Transformers has changed the picture. Most implementations apply transfer learning on a Trans- former model previously pre-trained on general NLU, on one or more languages, to the particular datasets and for predefined task.
3.1 DAE: Denoising Autoencoder phase
This new paradigm has guided the search for the best practice in each of the trending areas such prompting strategies (Brown et al., 2020), Few Shot Learning (Wang et al., 2021b), meta-learning (Finn al., 2017), also some concrete tasks like Intent Detection (Qin al., 2019), or noisy data (Siddhant Garg, 2021). Here, we present task- agnostic approach, outperforming some compet- itive methods for their corresponding tasks. should also be noted that our method benefits from
The Denoising Autoencoder is the first of the pro- posed 3-Phase approach, shown in Fig. la. Like any other Autoencoder, this model consists of an encoder and a decoder, connected through bottle- neck. We use two Transformer models as encoders and decoders simultaneously: RoBERTa (Liu et al., 2019), and all-Minni..M-I112.. (Wang et al., 2020). The underlying idea is that the bottle- neck represents the input according to the general
3

---

Dataset
Train DAE
Train SCL
Test
step. Each model's hidden size has been chosen its bottleneck size (768 in the case of RoBERR ER))). The key point of the first phase is to adapt the embedding representation of the encoder to the target dataset distribution, i.e., this step shifts the distribution from the general distribution learned by the pre-trained Transformers encoder into one of the target datasets. It should be noted here that this stage is in no way related to the categories for qualification. The first phase was implemented using the SBERT library.
SST2
69170
67349
872
SNIPS
262
262
65
SNIPS2
13084
13084
700
SST5
8544
8544
2210
AGNews
120K
120K
7600
IMDB
25K
25K
25K
Table I: Statistics for Train, Validation and Test dataset splits.
Dataset
Avg. Length
Max. Length
3.2 CL: Contrastive Learning phase
SST2
9
52
The second stage will employ Contrastive Learn- ing, more precisely, a Siamese architecture with co- sine similarity loss. The contrastive techniques are based on comparing pairs of examples or anchor- positive-negative triplets. Usually, these methods have been applied from semi semi- supervised point of view. We decided on supervised outlook where the labels to train in supervised manner are the categories for classification, i.e., we pick random example. Then, we can get a negative input by sam- pling from the other categories or a positive one by sampling from the same category. We combine this process of creating pairs with the imbalance correction explained below to get pairs of vector outputs (u, v). Given two inputs & and and label labely, based on their class similarity.
SNIPS
9
20
SNIPS2
9
35
SST5
19
56
AGNews
37
177
IMDB
229
2450
Table 2: Average and max lengths for each of the datasets mentioned in the paper.
distribution of the whole dataset. A balance needs to be found between preventing the information from being memorised and having sufficient sensi- tivity to be reconstructed by the decoder, forcing the bottleneck learn the general distribution. We add noise the Autoencoder to prevent the bot- tleneck from memorising the data. We apply a Dropout on the input to represent the noise. For- mally, for a sequence of tokens X {=o. ... ,., coming from a data distribution D, we define the loss as
1, if u and are in the same class. labely, 0. otherwise.
(2) We straightforward extension for the hier- archical dataset (AGNews News) by normalising these weights along the number of levels. In the case of two levels, we assign 1 for the case where all the labels match, 0.5 when only the first one matches, and 0 if none. After applying the encoder, we ob- tain i and G. We define the loss over these outputs as the Mean Squared Error (MSE):
LDAE = ED[log Xx(
(1)
where X is the sequence X after adding the noise. To support with example, masking token at the position would produce X 1..41.....27 The distribution
of Po in this Cross-Entropy loss corresponds to the composition of the decoder with the encoder. We consider the ratio of noise like another hyper- parameter to tune. More details can be found in Appendix 1, and the results section 5. Instead of upplying the noise to the dataset and running few epochs over we apply the noise on the fly, getting a very low probability of repeated input.
LCL = ||labely, Cosine e))l (3)
We apply this Contrastive Learning the en- coder DAE, i.e., the encoder and the bottleneck from the previous step. Again, we add an extra layer to this encoder model, producing our follow- ing final embedding. We denote this encoder after applying CL over DAE as DAECL (Fig. 1b). Sim- ilarly, we chose the hidden size per model as the
Once the model has been trained, we extract the encoder and the bottleneck, resulting in DAE (Fig. la), which will be the encoder for the next
Phttps: //www/ sbert. net www./eeee/iiiiiieeeeeemm
4

---

embedding size. CL plays the role of soft clustering for the embedding representation of the text based on the different classes. This will make fine-tuning much easier as it will be effortless to distinguish representations from distinct classes. The second phase was also implemented through the SBERT library.
function 4 gives f[1) matratio for one example. so we will never get something bigger than the maximum. The shape of this function similar to a negative log-likelihood, given a high ratio to the small classes and around the minimum for medium or bigger. A simple computation shows that the ratio 1 in function 4 is obtained at
3.2.1 Imbalance correction
(mazratio 1)/mazeatio
maa k
(6)
As mentioned in the previous section, we are us- ing a Siamese network and creating the pairs in a supervised way. It is a common practice to aug- ment the data before applying CL. In theory, we can make many example combinations as possible. In the case of the Siamese models, we have a total of n!! unique pair combinations that correspond to the potential number of pairs based on symmetry. Typically, data can be increased as much as one considers suitable. The problem that one may fall short or overdo it and produce overfitting in the smaller categories. Our augmentation is based on two pillars. We start with correcting the imbal- ance in the dataset by selecting underrepresented classes in the dataset more times but without re- peating pairs. Secondly, on the classes that have been increased, we apply noise on them to provide variants that are close but not the same to define more clearly the cluster to which they belong.
or mar 0.75 in our default case.
We duplicate this balanced dataset and shuffle both create the pairs. Since many combinations exist, we only take the unique ones without repeat- ing pairs, broadly preserving the proportions. Even so, just a few examples represent one class, the clustering process from CL can be affected by the augmentation because the border between them would be defined just for few examples. To avoid this problem, add a slight noise to the tokens when the text length is long enough. This noise consists of deleting some of the stop-words from the example. Usually, this noise was added to ere- ate positive samples and produced some distortion to the vector reprosentation. Adding it twice, in the augmentation and the CL, would produce too much distortion. However, since we are using a supervised approach, this does not negatively affect the model, as shown in the ablation section 5.
We balance the dataset by defining the number of examples that we are going to use per class based on the most significant class (where maæ; is its size) and range of ratios, from minyatio the mini- mum and maTratto the maximum. These upper and lower bounds for the ratios are and we chose 4 and 1.5 as default values. Formally, the new ratio is defined by the function:
3.3 FT: Fine-tuning phase
Our final stage is fine-tuning, obtained by employ- ing DAECL Fig 1b). We add a two-layer MLP on top classifier. We tried both to freeze and not to freeze the previous neurons from DAECL.
base model (as indicated in
As the final activation, we use Softmax, which is a sigmoid function for the binary cases. More formally, for K classes, softmax corresponds to
(mark) T
maITratio
f(x) = llog
(4)
log maxy
where x refers to the initial size of given class. After adding a lower bound for the ratio, we get the final amount
(21)
for i = 1,2, K (7)
Ej=1
As a loss function for the classification, we min- imize the use of Cross-Entropy:
newratios min min, f(class)) (ratio
(5a)
K
LFT
log(pk)
(8)
newclasss = newration x classk
(5b)
where P& is the predicted probability for the class and yk for the target. For binary classification datasets this can be further expanded as
for k 1,2, K in classification problem with classes, where class; and newclass; are the cardinalities of the class k. before and after the resizing, respectively. As we can observe, the
L = - log()) + (1 - y) log(1 - p)) (9)
5

---

3.3.1 Joint
with long inputs for binary classification in sentiment analysis. The length and size of this data made us consider only the RoBERTo as the base model.
We wanted to check if could benefit more when combining losses, i.e. by creating a joint loss based on the first and the second loss, and 3), respec-
tively.
We used Hugging Face API to download all the datasets apart from the second version of SNIPS In some cases, there was no validation dataset, we used 20% of the training set to create the vali- dation dataset. There was an unlabelled test set SN I PS, so we extracted another 10% for testing. The unlabelled data was used for the first training phase, not for the other ones. The first and second phases did not use the test set. We selected the best model for each of them based the results of the validation dataset. The test set was used the end of phase 3.
(10) This training was obtained as a joint training of stages one and two. By adding the classification head, like in the previous section, for fine fine-tuning, we got the version we denote as Joint (see Table 3).
Loint - DAA + LCL
4 Experimental Setup
The datasets for the experiments, the base mod- els used and the metries employed are detailed be- low.
4.1 Datasets
We have chosen several well-known datasets to carry out the experiments:
4.2 Models and Training
We carried out experiments with two diffcrent mod- els, small model for short text all-M MMMinMi L12-v2 (Wang et al., 2020)?, more concretely, version fine-tuned for sentence similarity and medium size model RoBERTa-bian (Liu et al., 2019)* which we abbreviate as RoBERT The ranges for the hyper-parameters below, as well as the values of the best accuracy, can be found in Appendix A.I.
Intent recognition on SNIPS (SNIPS Nat- ural Language Understanding benchmark) (Coucke al., 2018). For this dataset, we found different versions, the first one with just 327 examples and 10 categories. This one was obtained from the huggingface library", which shows how this procedure performs on small datasets.
We use Adam as the optimiser. We test differ- ent combinations of sapee-pprameters, subject to the model size. We tried batch sizes from 2 128, whenever possible, based on the dataset and the base model. We tested the encoder with both frozen and not frozen weights almost always get- ting better results when no freezing is in place. We tested two different truncations' lengths based ei- ther on the maximum length in the training dataset plus 20% or the default maximum length for the inputs in the base model. We tried to give more importance
The second version for SNIPS from K Kaggle* containing more samples and split into 7 classes that we call SNIPS2 is the most common one.
The third one is commonly used for slot pre- diction (Qin et al., 2019), although here we only consider task intent recognition. We used SST2 and SST5 from (Socher al., 2013) for classification containing many short text examples.
phase over the other by applying data augmentation in CL or extending the num- ber of epochs for the Autoencoder (Appendix A.1), We increased the relevance of the first phase by augmenting the data before applying random noise instead of changing the number of epochs. We tune the value of ratio, getting 0.6 as the most common best value. To increase the relevance of the second phase, we increased the number of inputs by cre- ating more pair combinations. Since we increased
We add AGNews (Zhang et al., 2015) to our list, medium size dataset that shows method over long text.
We complement the experiments with IMDB6 (Maas et al., 2011), big dataset
http://gggggnaaaeeecc//aaaaaaaaa built. in_intents
httlp:://www..e httt:://www..kggggecccm///aatttt
atis-snips htcross//hggiinaaeee...//ttttttt
/https://hugginggfcce wwweeeeenneeransfrrreeeeeeeeeeee all-MiniLM-L112-2
news
"https://hugginggfccecc http://gg.....cccaaattttss/////
"https://huggingfaat
http://ggggggaac...oorrrerrrerae co/roberta-bast
6

---

Dataset
3-Phase
Joint
FT
S-P
EFL
CAE
Self-E STC-DeBERTa FTBERT
RRBERTa
SNIPS
99.81
94.92
91.01
99.0
98.3
SNIPS2
98.29
98.0
97.57
99.0
98.3
SST2
95.07
93.12
90.28
96.9
94.78
SST5
56.79
53.88
52.27
56.2
AGNews
95.08
94.82
92.47
86.1
95.20
IMDB
99.0
95.07
91.0
96.1
all-Mini.MM-A12-00
SNIPS
100.00
91.98
92.89
99.0
98.3
SNIPS2
98.57
98.57
93.86
99.0
98.3
SST2
93.89
90.04
88.21
96.9
94.78
SST5
54.77
52.25
49.24
56.2
AGNews
94.83
94.28
89.57
86.1
95.20
Table Performance accuracy on different datasets using RoBERTa all-M 1.M-1111- models %. 3- 3-Phase refers main stages approach, while Joint denotes whose loss is based on the combination of the first two losses, and FT corresponds to the fine-tuning, We also add some SOTA results from other papers: S-P denotes Stack-Proppe Pruuk-Propagation (Qin et al., 2019), Self-E used to denote (Sun et al., 2020b) (in this case we chose the value from R..BERTaabbr a-base for a fair comparison), CAE is used to detonate (Phuong et al., 2022), STC-DeBERTa refers to (Karl and Scherp, 2022), EFL points to (Wang et al., 2021b) (this one uses RoBERTa-Large and FTBERT (Sun al., 2020a)
or decreased the relevance of one or other phases based on the amount of data, we used the same learning rate for the first two phases.
the SNIPS dataset with all-MimiLMM-11--1 We apply these three implementations to both base models, except IMDB, the input length is too long for all- all-MiniLM--12-- FT method on its performs the worst concerning the other two counterparts for both the models tested. Eventually, we may conclude that the 3- 3-Phase approach is generally the best one, followed by Joint, and as ex- pected, FT provides the worst. We can also observe that Joint and FT tend to be close in datasets with short input, while AGNews gets closer results for 3- 3-Phase and Joint.
4.3 Metries
We conducted experiments using the datasets pre- viously presented in Section 4.1. We used the stan- dard macro metrics for classification tasks, i.e., Ac- curacy, Fi-score, Precision, Recall, and the confu- sion matrix. We present only the results from Ac- curacy in the main table compare against other works. The results for other metrics be found in Appendix A.2.
We did not have a bigger model for those datasets with long input, pare against approaches with similar base models. We truncated the output for the datasets with the longest inputs, which may reflect smaller values our case. Since the advent of (Sun et al., 2020a), several current techniques are based on combining different datasets in the first phase through multi- task learning and then fine-tuning each task in de- tail. Apart from (Sun et al., 2020a), this is the case for the prompt-base procedure from EF (Wang et al., 2021b) well. Our method focuses ob- taining the best results a single task and dataset. Several datasets to pre-train the model could be used as a phase before the others. However, we
tried com-
5 Results
We assess the performance of the three methods against the several prominent publicly available datascts. Thus, here we evaluate the 3-Phase pro- cedure compared to Joint and FT approaches. We report the performance of these approaches in terms of accuracy in Table 3
We observe noticcable improvement of at least 1% with 3-Phase as compared the sec- ond best performing approach, Joint, in almost all the datasets. SNIP52 with all-Mini L.M L12-u2 we get the same values, while in IM DB with RoBERT we get a gap of point and 8 for
7

---

Dataset
Base Model
3-Plase
Joint
DAE+FT
CL+FT
Extra Imb.
No Imb.
FT
SNIPS
all-MiniLM-L12-U
100.00
91.98
98.05
99.81
99.92
99.88
92.89
SNIPS2
MU-MinLLM-L12-11
98.57
98.57
94.14
97.86
98.00
97.85
93.86
SST2
RoBERTa
95.07
93.12
83.72
94.72
94.50
94.04
90.28
SST5
RoBERTa
56.79
53.88
46.06
56.52
56.24
55.84
52.27
AGNews
aoBERTa
95.08
94.82
91.53
95.24
95.01
94.26
92,47
IMDB
RoBERTa
99.00
95.07
93.00
94.86
97.00
94.76
91.10
Table 4: Ablation results. As before, 3-Phase, Joint, and FT correspond the stages approach, joint losses, and Fine-tuning, respectively. Here, DAE+FT denotes the denoising autoencoder together with fine-tuning. CL+FT denotes the contrastive Siamese training together with fine- fine-tuning, Imb. means 3haas but skipping the imbalance correction, and Extra Imh. refers to increase of the imbalance correction to a millyatio- and mazratto=20.
doubt the possible advantages of this the first and second phases would negatively affect this learning, and those techniques focused on training the clas- sifier with several models would negatively affect the first two phases.
where Extra Imb. increases the upper bound for the ratio and No Imb. excludes the imbalance method. Both cases generally produce lower accuracies than 3- Phase, being No Imb. slightly lower. The last column corresponds fine-tuning FT.
To complete the picture, CAE (Phuong et al., 2022) is an architecture method, which is also independent of the pre-training practice. A self- explaining framework is proposed in (Sun et al., 2020b) architecture method that could be implemented on top of our approach.
All these experiments proved that the proposed 3- Phase approach outperformed all the steps inde- pendently, on its or combined the Denoising Autoencoder and the Contrastive Learning loss.
Conclusion
5.1 Ablation study
The work presented here shows the advantages of fine-tuning model in different phases with an imbalance correction, where each stage considers certain aspects, either as an adaptation to the text characteristics, the class differentiation, the imbal- ance, or the classification itself. We have shown that the proposed method can be equally or more effective than other methods explicitly created for a particular task, even if we do not use auxiliary datasets. Moreover, in all cases, it outperforms classical fine-tuning, thus proving that classical fine-tuning only partially exploits the potential of the datasets. Squeezing out all the juice from the data requires adapting the data distribution and grouping the vector representations according the task before the fine-tuning, which in our case is targeted towards classification.
We conducted ablation experiments on all the datasets, choosing the same hyper-parameter and base model as the best result for each one. The results can be seen in Table 4.
We start the table with the approaches men- tioned: P-hhscandd Joint. We wanted see combining the first two phases could produce better results as those would be learned simultaneously. The results show that merging the two losses al- ways leads to worse results, except for one of the datasets where they give the same value.
We start the ablations by considering only DAE right before fine-tuning denoting it as DA In this case, we assume that the fine-tuning will carry all the class information. One advantage of this outlook is that still applies to models that employ a small labelled fraction of all the data (i.e., the unlabelled data represents the majority). The next column, CL+FT, replaces DAE with Contrastive Learning, concentrating the attention the classes and not the data distribution. Considering only the classes and fine-tuning in CL+FT, we get better results than in DAE+ FT, but still lower than the 3- Phasein almost all the datasets. Right after, we add two extreme cases of the imbalance correction,
7 Future work
The contrastive training phase benefits of data aug- mentation, i.e., we can increase the number of examples simply through combinatorics. How- ever, this can lead to space deformation for small datasets, even with the imbalance correction, as fewer points are considered. Therefore, overfitting
8

---

occurs despite the combinatoric strategy, Another advantage of this phase balancing the number of pairs with specific values. This practice allows us, for example, to increase the occurrence of the underrepresented class to make its cluster as well defined as those of the more represented categories (i.e. ones with more examples). This is a partial solution for the imbalance problem.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, and Dario Amodei. 2020. Language models are few-shot learners.
In the future, we want to address these problems. For the unbalanced class in the datasets, seek so- lution to avoid overfitting to the under-represented classes and extend our work to support a few shot learning settings (L..). To do so, we are going to analyze different data augmentation techniques. Among others, Variational Autoencoders. Recent approaches for text generation showed that hierar- chical VAΣ models, such as stable diffusion models, may produce very accurate augmentation models. One way to investigate this is to convert the first phase into a VAE model, allowing us to generate more examples from underrepresented classes and generally employ them all in the FSL setting. Finally, we would like to combine our approach with other fine-tuming procedures, like prompt methods. Adding a prompt may help the model gain previous knowledge about the prompt struc- ture instead of learning the prompt pattern simulta- neously during the classification while fine-tuning.
S. Chopra, R. Hadsell, and Y. LeCun. 2005. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recog- nition (CVPR 005, volume I, pages 539-546 vol. 1.
Alexis Conneau and Douwe Kiela. 2018. SentEval: An evaluation toolkit for universal sentence representa- tions. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (IRRCC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Calt- agirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. Snips voice platform: an embedded spoken language understanding system for private- by-desiga voice interfaces.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing.
Limitations
Chelsea Finn, Pieter Abbeel, and Sergey Levine. 2017. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th Interna- tional Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1126-1135 PMLR.
This approach is hard to combine with other fine- tuning procedures, mainly those which combine different datasets and use the correlation between those datasets, since this one tries to extract and get close as possible to the target dataset and task. The imbalance correction could be improved, restricting to cases where the text short because it could be too noisy or choosing the tokens more elaborately and not just stop words. It would be necessary to do more experiments combined with other approaches, like the prompt base, to know if they benefit from each other or if they could have negative repereussions in the long term.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence em- beddings. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Process- ing. pages 6894-6910, Online and Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. 2015. Made: Masked autoencoder for distribution estimation. In Proceedings of the 32nd International Conference on International Confer- ence on Machine Learning Volume 37, ICML 15, page 881 188-. JMLR.org.
Acknowledgments
The authors would like to thank the members of the AIApps Research Group in the Huawei Ireland and London Research Centers for their valuable discussion and comments. We especially want to thank Milan Redzic, Tri Kurniawan Wijaya and Jinhua Du for their help.
Michael Gutmann and Aapo Hyvärinen. 2010. Noise- contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the Thirteenth International Conference on Artifi- cial Intelligence and Statistics, volume 9 of Proceed- ings of Machine Learning Research, pages 297-304, Chia Laguna Resort, Sardinia, Italy. PMLR.
9

---

Fabian Karl and Ansgar Scherp. 2022. Transformers are short text classifiers: study of inductive short text classifiers on benchmarks and real-world datasets.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng. and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised Contrastive Learning. arXiv e-prints, page arXiv:2004.1 1362.
Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, volume 29,
Mark A. Kramer. 1991. Nonlinear principal compo- nent analysis using autoassociative neural networks. Aiche Journal, 37:233-243.
Curran Associates, Inc.
Ruizhe Li, Xiao Li. Chenghua Lin, Matthew Collinson, and Rui Mao. 2019. A stable variational autoen- coder for text modelling. In Proceedings of the 12th International Conference on Natural Language Gen- eration, pages 594-599, Tokyo, Japan. Association for Computational Linguistics.
Chi Sun, Xipeng Qiu, Yige Xu, and Xuanjing Huang. 2020a. How to fine-tune bert for text classification?
Zijun Sun, Chun Fan, Qinghong Han, Xiaofei Sun, Yuxian Meng, Fei Wu, and Jiwei Li. 2020b. Self- explaining structures improve nlp models.
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. ArXiv, Abs/1907.11692.
Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Fierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: I Learning useful rep- resentations in deep network with local denoising criterion. J. Mach. Learn. Res., 11:3371-3408.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Lin- guistics.
Kexin Wang. Nils Reimers, and Iryna Gurevych. 202 la Tsdae: Using transformer- sequential denoising auto-encoder for unsupervised sentence embedding learning.
Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021b. Entailment as few-shot learner.
Wenhui Wang. Furu Wei, Li Dong. Hangbo Bao, Nan Yang, and Ming Zhou. 2020. Minilm: Deep self- attention distillation for task-agnostic compression of preitrdinee transformers. In NeurlPS 2020. ACM.
Yishu Miao, Lei Yu, and Phil Blunsom. 2015. Neural variational inference for text processing.
Nguyen Minh Phuong. Tung Le, and Nguyen Le Minh. 2022. Cae: Mechanism to diminish the class imbal- anced in slu slot filling task. In Advances in Com- putational Collective Intelligence, pages 153-163, Cham. Springer International Publishing.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization.
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun, 2015. Character-level convolutional networks for text clas- sification. In NIPS.
Libo Qin, Wanxiang Che, Yangming Li, Haoyang Wen, and Ting Liu. 2019. stack proo-propaggttoon framework with token-level intent detection for spoken language understanding. pages 2078-2087.
Nils Reimers and Iryna Gurevych. 2019, Sentence-bert Sentence embeddings using siamese nert-networks.
Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. 2021. Step- unrolled denoising autoencoders for text generation.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. 2015 IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), pages 815-823.
Varun Thumbe Siddhant Garg, Goutham Ramakrish- nan. 2021. Towards robustness to label noise in text classification via noise modeling. arXiv preprint urXiv:2101.122145:
10

---

A Appendix
A.1 Hyper-parameters
This appendix section shows the final hhper-paramenee parameters from the best results in Table 3. The column at the end on the right contains the search space used for training. Some of the values were not used for training, either because of computational limitations because they were not realistic for some datasets.
Dataset Best Hyper-parametens
Parameter
SNIPS
SNIPS2
SST2
SSTS
AGNews
IMDB
SearchSpace
RoBERTa
Learning rate DAE
5e-5
5e-5
le-5
le-5
2e-4
le-5
[1e-4, 2e-4, 1e-5. 1e-6)
Leaming rate CL
5c-5
5e-5
le-5
le-5
2e-5
le-5
[Ie-4, 2c-4, le-5. 1o-6)
Learning rate FT
le-5
le-5
le-5
le-5
20-5
le-5
[le-4, 2e-5, lc-6)
Epochs DA
(2, 3,4, 12]
Epochs
12
3,44, 12]
Max Epochs FT
70
70
70
70
70
50
[20, 50, 70)
Batch size DAE
32
32
64
64
16
6
(2.68.12.11.33.64412888
Batch size CL
32
32
64
64
16
6
12.68811.116,22.64,222
Batch size FT
32
32
64
64
16
6
12,68,1,2,1,,3,,444,13,,,,
Eps in F7
2c-5
2e-5
20-5
20-5
20:5
20-5
[2c-05)
Use Length*
48**
108*1
512
512
500*
{*), +1 215)
Freezing Encoder
False
False
False 0.6
False 0.5
True
False
(True, False]
Deleting ratio*
0.6
0.7
0.6
0.6
(0.3, 0.4, 0.5. 0.6, 0.7)
all-MinLLM -L1--22
Learning DAE
5e-5
le-5
le-5
Se.5
Se:5
[le-4, 2e-4, Se 1e-5, le-6) [ee-4, 2o-4, Se-5, le-5, le-6)
Leaming rate CL.
5c-5
le-5
le-5
5e-5
5e-5
Learning FT
5e.5
le-5
le-5
le.5
le.5
[le-4, le-5, 2e5,, 1e-6)
Epochs DA
[2, 3, 12)
Epochs
12
3,44, 12]
Max Epochs 2
70
70
70
70
70
[20, 50, 70}
Batch size DAE
32
32
32
128
32
12,,8,12211,322,,112888)
Batch size CL
32
32
32
128
32
12.6,,,12.16,22,44,1122
Batch size FT
32
32
32
128
32
12,6,,11,,,,,,,,,,,,14,,,,
Eps in F7
2e:5
2e-5
20-5
2e-5
20:5
[2c-05)
Use Length
108*4
$12
108++
512
[*3, * *,212
Freezing Encoder
False
False
False
True
(True, False)
Deleting ratio*
0.6
0.3
0.6
0.7
0.6
(0.3. 0.4, 0.5, 0.6, 0.7)
Table 5: Hyper-parametent configurations and search space of the experiments." means these not real epochs since the input data is not always the same. The data was masked on the fly; therefore, each epoch differs. *2 We used early stopping approach for the FT phase. +3 We only consider the epsilon hyperparameter in the AdamW optimizer for FT. The other two phases use the default value from the Transformers library (le-06). #4 This hyper-p parameter was estimated initially with the training dataset with large margin. This was applied for datasets with very short sentences, like SNIPS. This hyper- parameter estimates the length of the sequences using the 10% of the examples. This estimation is multiplied by 1.2 and is added as the maximum size of the sequences for embedding layers. The difference is that it was done on the fly and not preserved in this case.
hyper-parameter

---

A.2 Other metrics
this section of the appendix, we present the results obtained for metries other than accuracy. More specifically, we present three tables: Precision and Recall in Table 6, and FI (Table 7). These metrics show better the role played by the imbalance correction. The notation follows the Table 3.
Precision
Recall
Dataset
3- 3hPhase
Joint
FT
3-Phase
Joint
FT
RoBERTa
SNIPS
99.81
94.94
91.04
99.81
94.99
91.03
SNIPS2
98.26
97.96
97.50
98.30
98.09
97.72
SST2
95.62
94.29
92.53
95.05
92.32
89.21
SST5
55.51
53.23
49.50
53.71
51.18
49.02
AGNews
95.09
94.82
92.44
95.08
94.82
92.47
IMDB
98.08
95.56
91.83
100.0
96.74
91.03
allMMinLL--12-1
SNIPS
100.00
92.05
93.12
100.00
92.06
92.98
SNIPS2
98.61
98.64
93.89
98.68
98.60
94.14
SST2
93.92
92.32
88.79
94.89
91.24
88.13
SST5
59.57
61.62
53.15
49.02
46.02
40.02
AGNews
94.84
94.28
89.57
94.83
94.28
89.57
Table Precision and Recall values. The best values are shown bold
Dataset
3-Phase
Joint
FT
S-P
EFL
CAE
FTBERT
RoBERTa
SNIPS
99.81
94.95
91.03
97.0
97.0
SNIPS2
98.28
98.01
97.57
97.0
97.0
SST2
95.14
93.29
90.82
SST5
54.59
52.18
49.24
AGNews
95.08
94.81
92.45
79.5
95.20
IMDB
99.03
95.10
91.43
all-MiniLA
M-L12-22
SNIPS
100.00
92.0
92.81
97.0
97.0
SNIPS2
98.63
98.60
93.90
97.0
97.0
SST2
94.39
91.78
88.46
SSTS
53.78
52.69
45.66
AGNews
94.83
94.27
89.56
79.5
95.20
Table 7: Fl values for the best results. The best values are shown in bold
12


Reviews:
[{"review_id": "xMkxVu4j2z", "reviewer": "Reviewer_oYVS", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your valuable comments and helpful feedback, which will be considered for the revised version of the paper.\n\nWe will address the minor errors and typos exposed for the final version. The ablation table contained two typos, one being the values exchange between SST2 and SST5 at the Joint column. This gave rise to the confusion where the value 92.74 stood out from the others.\n\nAccording to the reviewer's comments, we have simplified the experiments and computed Precision, Recall, F1, Accuracy and confusion matrices that will be included in the paper Appendix. Moreover, we will add some comparisons between these new metrics to facilitate the understanding of how the unbalance correction affects one of the datasets. We would like to explain that Accuracy was used to report the results since this is the standard metric for the different datasets used in previous works, so our performance can be compared to prior studies.\n\nWe conducted an extensive initial grid search using a reduced version of the datasets and found the two best learning rates proposed. We decided to omit this typical hyperparameter search to focus the reader's attention on the main experiments. However, we will describe the process of this grid search for each dataset in the corresponding section of the paper, mentioning all the values tested. Furthermore, we will make the code publicly available for reproducing the whole process.\n\nRegarding the paper page limitation, the authors followed the paper submission guidelines. It seems that the \"Limitations\" section does not count as part of the paper (see https://2023.emnlp.org/calls/main_conference_papers/#:~:text=will%20not%20count%20towards%20the%20page%20limit): \"This section will appear at the end of the paper, after the discussion/conclusions section and before the references, and will not count towards the page limit.\" Please let us know whether this information has changed, and we will process it accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method was explained well and was easy to understand. Also, the 2 additional phases before fine-tuning make complete sense and is an intuitive, yet simple, approach to increase the performance of the pre-trained model after fine-tuning. Also, ablation studies were decent and proving that the phases add value to the final fine-tuning.", "reasons_to_reject": "* Since the authors mention the unbalanced datasets multiple times and propose an approach to tackle it, \"accuracy\" was not the best metric to report, specifically since they mention they computed precision, recall, and F-1 score as well (or they could be added to the appendix).\n\n* Moreover, although some hyper-params, e.g. value of min and max ratios, where thoroughly searched over, but some other important hyper-params, e.g. the learning rates or larger max number epochs, where not searched and experimented.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors propose a new multi-phase approach for fine-tuning transformers to a specific dataset. The first phase adapts the pre-trained transformer to the goal dataset distribution through a DAE (unsupervised). The second phase applies a soft clustering through using a  siamese architecture with a cosine similarity loss using positive and negative examples. Moreover, during the second phase they also apply their novel approach to correct for the unbalanced class sizes in the data. Finally, after the first two stages, they apply the final fine-tuning to the goal dataset and show that the first two phases significantly improve the performance of the model on multiple public datasets.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "There are a couple minor errors and typos in the text. For example, in Table 4, row SST5, column Joint I do not think the accuracy is 92.74 while the others are under 60. Or in the same table in the caption mentions \"man_ratio\" whereas that should be \"max_ratio\"."}}, {"review_id": "qW5BGFVhOJ", "reviewer": "Reviewer_Nkv3", "summary": "", "questions": "Question A: Equation 3. Do I understand it right: it is just the modulus of the difference of two numbers?\nQuestion B: I suggest using bold font in table 3 to highlight the best result values.\nQuestion C: In table 4, column \"Joint\", rows \"SST2\" and \"SST5\": are the values correct? (why are the values not the same as in the table 3)\nQuestion D: Why did you decide to perform no ablation studies for the IMDb dataset?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, we would like to thank you for your valuable comments and the useful feedback which will be considered for the revised version of the paper. \n\nWe have taken into consideration the reviewer's comments in the \"Typos, Grammar, Style and Presentation Improvements\" section with detailed feedback: \n1. The Title has been corrected now matching the one from the submission.\n2. The abbrevations have been definded at the first ocurrence.\n3. The symbol \"~\" has been added before each \"\\ref\".\n4. The loss symbol has been standarized to \"\\mathcal{L}\" in all the equations.\n5. It has been clarified in the text that the domain of the variable \"x\" is any possible size of a category, meaning an integer value between the size of the smallest category and the biggest one.\n6. Replaced \"min\" by \"\\min\" in all the ocurrences.\n7. The problem from \"DAECL\" may be related to formatting issues from the submission web site. We do not see the error in our version, neither can find it in the pdf from the rebuttal. We will ensure this does not happens in later versions, probably presenting the middle models names in italic.\n8. We have deleted the empty line before any equation to correct the extra space.\n9. \"please\" deleted.\n10. Missing footnotes for the datasets have been added including \"\\footnote{\\url{https://huggingface.co/datasets/snips_built_in_intents}}\" for snips.\n11. We are standarizing the text to present simple.\n12. The format of \"1e-5\" and any other scientific notation have been corrected.\n13. We have standarized to \"3 - Phases\" all the text.\n14. \"where each takes\" has been replaced by \"where each stage takes\".\n15. We replaced \"complicated\" with \"hard\" in line 572, as it suits best.\n16. This has been clarified with \"between those datasets\".\nApart from the errors pointed out by the reviewer, a detailed revision of the text will be included in the final version to solve other possible minor errors.\n\nQuestion A: Yes, the reviewer understood it correctly. We used an equation to explain it better instead of writing it in the text. The description of the loss function we used can be found in the SBERT library:\nhttps://www.sbert.net/docs/package_reference/losses.html?#cosinesimilarityloss\n\nQuestion B: The authors found this suggestion useful and have highlighted the best values in bold.\n\nQuestion C: As the reviewer pointed out, there were two typos when adding the values to the table. The first one: in the column Joint, the values between SST2 and SST5 had been interchanged, but now they have been corrected. The second one is the value \"92.74\", which should be \"92.47\" because of a typing error in which the order of the decimals was reversed.\n\nQuestion D: IMDB is the dataset that takes longer because of its size and the inputs' length, forcing us to use a small batch size. At first, we decided to perform the ablation experiments with smaller datasets, which could provide enough evidence for our claims. However, we have carried out the experiments in the IMDB dataset as well for completeness, and the following final results will be included in the last version of the paper (including typo correction):\n\nDataset | Base Model | 3 Phases | Joint | DAE+FT |CL+FT |Extra Unb. |No Unb. |   FT\n\nSNIPS       allMiniLM     100.00     80.15     98.05      99.81      99.92        99.88      77.85\n\nSNIPS2     allMiniLM      98.57      96.13     94.14      97.86      98             97.85       93.74\n\nSST2         RoBERTa     95.07      92.47     83.37      94.72      93.80         93.80      90.28\n\nSST5          RoBERTa    56.79      53.88     46.06      55.70      55.79         55.84      52.27\n\nAGNEWS   RoBERTa    94.99      93.37     91.53      95.18      94.30         94.18      90.59\n\nIMDB          RoBERTa   93.39      95.07     90.63      93.51       93.74        93.19       87.87   \n\n\nTo facilitate the reproducibility of the experiments we will make the code publicy available, including data preprocessing, and the training loops to replicate the experiments.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Reasons To Accept:\n* New fine-tuning algorithm outperforming the classical fine-tuning.\n* New unbalance correction method used by the authors during one of the fine-tuning stages.\n* Careful explanation of the ideas behind each stage of the proposed algorithm: how it helps to better exploit the data during fune-tuning (why the algorithm outperforms the classical fine-tuning).\n* Task-independence of the proposed approach (it can be used to fine-tune a base model for different tasks, not only classification).\n* Extensive experiments with many models and datasets.\n* Ablation study.", "reasons_to_reject": "Aside from minor linguistic errors (which are not difficult to correct), I do not see much reason to reject the paper.", "questions_to_authors": "Question A: Equation 3. Do I understand it right: it is just the modulus of the difference of two numbers?\nQuestion B: I suggest using bold font in table 3 to highlight the best result values.\nQuestion C: In table 4, column \"Joint\", rows \"SST2\" and \"SST5\": are the values correct? (why are the values not the same as in the table 3)\nQuestion D: Why did you decide to perform no ablation studies for the IMDb dataset?", "paper_topic_and_main_contributions": "The paper explores the problem of adjusting a (large) pretrained model to a new supervised classification task. Specifically, the authors propose a new three-staged fine-tuning algorithm: denoising autoencoder training, supervised contrastive learning, and finally fine-tuning.\n\nContributions:\n* New fine-tuning algorithm outperforming the classical fine-tuning.\n* New unbalance correction method (used by the authors during one of the fine-tuning stages).\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* The paper title differs from the one here in OpenReview (no \"How to\")\n* Lines 98,99: \"NLU\", \"FNN\". Missing decryption on first use.\n* Line 237. \"Figure 1\" splits on two lines. Use unbreakable space.\n* Line 278 (eq 3). The symbol for the loss differs from that of eq. 1 (L vs \\mathcal L)\n* Eq. 4. Not quite clear what \"x\" means.\n* Line 321. Use \\min for \"min\" (\\mathrm font should be here, not italic).\n* Line 345 (and other places). \"DAECL\". Maybe I am wrong, but currently it looks like $DAECL$ (inline math). I believe just italic here would be fine.\n* Eq. 8 and eq. 9: Top space between the equation and text is bigger than bottom space.\n* Line 369. No need for \"please\".\n* Line 380. Provide a link to Huggingface.\n* Line 421. \"we used\" (past simple), \"we test\" (present simple).\n* Line 444. Formatting of \"1e-05\" is not correct (excess spaces around minus sign).\n* Line 456: Sometimes it \"3 - Phases\", sometimes \"3 Phases\", sometimes \"3Phases\" (different symbols for one notion)\n* Line 576. I believe better say \"where each stage\".\n* Line 572: I think that \"hard\" better suits here than \"complicated\".\n* Line 594: \"between\" what?"}}, {"review_id": "ZY8NdyGzsS", "reviewer": "Reviewer_j83o", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer, thank you for your valuable comments and helpful feedback, which will be considered for the revised version of the paper. Specifically, we have changed the misplaced results in Table 4 and will rewrite the introduction section for clarity, separating the motivation and the related work.\n\nThe reviewer claims the work lacks some novelty by combining existing approaches. First, we would like to remark that this specific setup has not been built in the literature yet. Moreover, we conducted an extensive study showing why the presented model is the most natural way to combine these methods. We hypothesized that differencing the proposed fine-tuning phases and utilizing them in that specific order (unsupervised->un. correction+self-supervised->supervised) is most suitable for the data representation:\n\n\t1. On the whole, we considered that the DAE model should be in the first place because the model learns for each sample to be robust against noisy inputs (unsupervised approach).\n\n\t2. Subsequently, we decided to incorporate CL to learn the similarities between different samples in the same class and the dissimilarities with the other class examples (self-supervised approach). In addition, we proposed a novel unbalance correction to select more frequently the underrepresented classes while avoiding the repetition of inputs (un. correction).\n\n\t3. Finally, fine-tuning the model to the given dataset will adapt the representation for the predefined task (supervised approach).\n\nWe showed through the experiments that combining all the joint losses is not as good as having this specific configuration. Furthermore, we agree with the reviewer regarding the increased training cost of using this fine-tuning configuration. There is indeed an increasing time using these two previous fine-tuning phases. Still, we have demonstrated that the performance couldn't be that high by only fine-tuning the model in the classical way for the classification tasks, so testing this configuration is very valuable as long as the task goal is to increase the performance of the classifiers. Similarly to large language models, this training time could be reduced using higher computational power when there is any time constraint.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors provide extensive experimental results on several datasets to demonstrate their 3-stage finetuning method's performance.\nIt empirically shows the gain from DAE pretraining and contrastive pretraining. The total setup is detailed, ensuring reproducibility. It may provide a good empirical example for the community.", "reasons_to_reject": "Personally, I find myself not surprised enough by reading this paper. Adding a denoising pretraining phase will help, adding a contrastive pretraining phase will help, but it seems reasonable from my viewpoint. So it maybe somehow lacks novelty by just combining different existing techniques. I am willing to change my rating if I am persuaded that these are not simple A+B.\n\nAnother problem is that while these stages of pretraining help, what is the training cost? Are these costs meaningful for cost-performance trade-off? \n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper propose a three-phase fine-tuning approach to adapt a pre-trained base model to a supervised classification task, yielding more favourable results than classical fine-tuning, and propose an unbalance correction method.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Maybe the results from Table 4 (Joint method) are misplaced?\n\nI personally like the first paragraph of introduction, but the following writing can be improved to state clear what is your procedure and what are related work in intro."}}]

Author rebuttal:

Meta-review: This paper proposes a multi-stage adaptation technique (unsupervised+contrastive+supervised) to improve quality of transfered models. All reviewers find the experiments sound and the technique well-explained. Only concern is raised by Reviewer j83o related to the novelty of the work as individual stages are not novel. I believe combining different techniques and demonstrating the benefit has great value for the community.
Decision: Accept-Main