Cluster & Tune: Enhance BERT Performance in Low Resource Text Classification
Authors: Eyal Shnarch
Ariel Gera
Alon Halfon
Lena Dankin
Leshem Choshen
Ranit Aharonov
Noam Slonim
Venue:  (2021)
Keywords: low resource
BERT
clustering

Abstract: In data-constrained cases, the common practice of fine-tuning BERT for a target text classification task is prone to producing poor performance. In such low resources scenarios, we suggest performing an unsupervised classification task prior to fine-tuning on the target task. 
Specifically, as such an intermediate task, we perform unsupervised clustering, training BERT on predicting the cluster labels. We test this hypothesis on various data sets, and show that this additional classification step can reduce the demand for labeled examples.
We further discuss under which conditions this task is helpful and why. 

Under review as a conference paper at ICLR 2021
CLUSTER & TUNE: ENHANCE BERT PERFORMANCE IN TOPICAL TEXT CLASSIFICATION WHEN LABELED DATA IS SCARCE
Anonymous authors Paper under double-bling blind review
ABSTRACT
In cases where labeled data is scarce, the common practice of fine-tuning BERT for a target text classification task prone to producing poor performance. In such scenarios, we suggest performing an unsupervised classification task prior to fine- tuning on the target task. Specifically, as such an intermediate task, we perform unsupervised clustering, training BERT on predicting the cluster labels. We test this hypothesis on various data sets, and show that this additional classification step significantly reduce the demand for labeled examples mainly for topical classification tasks. We further discuss under which conditions this task is helpful and why.
INTRODUCTION
One of the most practical NLP use cases is the task of text classification, where the goal is to tomatically assign a new text instance into a subset of pre-specified categories. Text classification applications include topic detection, sentiment analysis, and spam filtering, to name just few exam- ples. The standard paradigm relies on supervised learning, where it is well known that the size and quality of the labeled data strongly impact the performance of the trained classifier. Hence, as with many other supervised learning tasks, developing a text classification scheme in practice, typically requires to make the most out of relatively small set of annotated examples.
The emergence of transformer-based pretrained language models such as BERT DDvlin et al. 2018 has reshaped the NI.P landscape, leading to significant advances in the performance of most NL.P tasks, text classification included (e.g., Nogueira & Cho 2019: Ein-Dor et al. 2020). These models typically rely on pretraining a transformer-base
neural network on massive and heterogeneous cor- pora general Masked Language Modeling (MLM) task, i.e. predicting word that is masked in the original text. Later on, the obtained model is fine-tuned to the actual task of interest, termed here the target task, using the labeled data available for this task. Thus, pretrained models serve as
general sentence encoders which can be adapted to a variety of tasks (Lacroix et al. 2019 Wang al. 2020a).
Our focus in this work is on a challenging yet common scenario, where the textual categories are trivially separated; furthermore, the available labeled data is scarce. There are many real-world cases in which data cannot be sent for massive labeling by the crowd (e.g.. due to confidentiality of the data, or the need for a very specific expertise) and the availability of experts is very limited. In such setting, fine-tuning overcome this, one may take a gradual approach composed of various steps. One possibility is to further pretrain the model with the self-supervised MI.M task over unlabeled data taken from the target task domain (Whang et al. 2019). Alternatively, one can train the pretrained model using a supervised intermediate task which is different in nature from the target-task, and for which labeled data is more readily available (Pruksachatkun et al. 20200 Wang et al.) 2019a Phang et al. 2018). Each of these steps is expected provide better starting point in terms of the model parameters for the final fine-tuning step, performed over the scarce labeled data available for the target task, aiming end up with improved performance.
pretrained model is expected to yield far from optimal performance. To
Following these lines, here we propose a simple strategy, that exploits unsupervised text clustering as the intermediate task towards fine-tuning a pretrained language model for text classification. Our

---

Under review as a conference paper at ICLR 2021
Pre-training
Inter-training
Fine-tuning
Intertrain using MLM
BERT
pretrain gemeral domain
BERT
tanges tack
Intertznin
BERT
etustaring
intectasin
Interte'sin using clustering
BERT
MLM
Figure I: BERT phases circles are training steps which produce models, represented rectangles. In the pre-training phase, only general corpora is available. The inter-training phase is exposed to target domain data, but not its labeled instances. Those only available at the fine-tuning phase.
work is inspired by the use of clustering to obtain labels for training deep networks in computer vision (Gidaris al., 2018; Kolesnikov et al. 2019). Specifically, we use an efficient clustering technique, that relies on simple Bag Of Words (BOW) representations, to partition the unlabeled training data into relatively homogeneous clusters of text instances. Next, we treat these clusters as labeled data for an intermediate text classification task, and train BERT with or without additional MLM pretraining - with respect to this multi-class problem, prior to the final fine-tuning over the actual target-task labeled data. Extensive experimental results demonstrate the practical value of this strategy on a variety of benchmark data, most prominently when the training data available for the target task is relatively small and the classification task is of topical nature. We further analyze the results to gain insights as to when this approach would be most valuable, and propose future directions expand the present work.
2 INTERMEDIATE TRAINING USING UNSUPERVISED CLUSTERING
A pre-trained transfer model, such BERT, is typically developed in consecutive phases. First, the model is pretrained over massive general corpora with the MLM task The obtained model is referred henceforth BERT. Second, BERT is finetuned in supervised manner with the available labeled examples for the target task at hand. This standard flow represented via Path-1 in Fig.
An additional phase can be added between these two, referred to next as intermediate training, or inter-training in short. In this phase, the model is exposed to the corpus of the target task, corpus of the same domain, but still has no access to labeled examples for this task.
A common example for such an intermediate phase is continue to intertrain BERT using the self- supervised ML. task over the corpus or the domain of interest, sometimes referred to as further or adaptive pre-training (e.g.. (Grurangan et al. 2020). This flow is represented via Path-2 in Fig. [ and the resulting model is denoted BERT MLM. standing for Intermediate Task: MLM.
BERT was originally also pretrained over a "next sentence prediction" task Devlin et al. 2018); however, later works such as Yang et al. 2019) and Ind[ii et al. 2019b) have questioned the contribution of this additional task and focused on ML..
2

---

Under review as a conference paper at ICLR 2021
Train
Dev
Test
classes
Topical
Yahoo! answers
15,000
3,000
3,000
10
Yes
DBpedia
15,000
3,000
3,000
14
Yes
CFPB
15,000
3,000
3,000
15
Yes
20 newsgroups
10,182
1,132
7,532
20
Yes
AG's news
15,000
3,000
3,000
4
Yes
ISEAR
5,366
766
1,534
Yes
SMS spam
3,900
557
1,115
No
Subjectivity
7,000
1,000
2,000
No
Polarity
7,463
1,066
2,133
2
No
CoLa
7,592
959
1,043
No
Datasets details: description, number of classes, the split train/dev/test sets and whether the dataset is topical or not (see details in text).
Table 1:
A key motivation of this paper is to propose new type of intermediate task, which is designed to be aligned with a text classification target task, and straightforward to use in practice. The underlying intuition is that inter-training the model over related text classification task, would be more beneficial compared to MLM inter-training, which focuses on different textual entities, namely predicting the identity of single token.
Specifically, we suggest unsupervised clustering for generating pseudo-labels for inter-training. be viewed as weak labels, but importantly they are not tailored require
These pseudo-labels a specific design per target task. Instead, we suggest generating pseudo-labels in a way independent of the target task. The respective flow represented via Path-3 in Fig. I this flow, first use unsupervised clustering to partition the training data into clusters. Next, we use the obtained par- tition as 'labeled' data a text classification task, where the classes are defined via the n, clusters, and intertrain BERT over these data to predict the cluster label. In line with MLM, inter-training includes a classifier layer on top of BERT, which discarded before the fine-tuning stage. The resulting inter-trained model is denoted IERTT.
Finally, Path-4 in Fig- [ represents a sequential composition of Path-2 and Path-3. Thus, this flow, we first intertrain BERT with the MI.M task. Next, the obtained model is further intertrained to predict the ne clusters, as in Path-3. The model resulting from this hybrid approach is denoted BERT MLM-
Importantly, following Path-3 or Path-4 requires no additional labeled data, and involves an a-priori clustering of training instances that naturally gives rise to alternative training task. As we show in the following sections, in spite of its simplicity, this strategy provides significant boost in classification performance in the common scenario, where labeled data for the final fine-tuning is in short supply.
additional inter-
3 EXPERIMENTS
3.1 TASKS AND DATASETS
We consider 10 datasets that cover variety of classification tasks and domains. Of these, topical datasets (including ISEAR which is somewhat borderline), and 4 are non-topical datasets. A topical dataset splits sentences by a high level distinction related what the sentence is about (e.g., sports vs. economics). On the other hand, non-topical datasets look for finer stylistic distinctions that may depend on the way the sentence is written or on fine details rather than on the central meaning it discusses. It may also separate almost identical sentences, for example, "no" could distinguish between sentences with negative and positive sentiment.
The datascts are: Yahoo! Answers (Zhang et al., 2015), which separates answers and questions to types; DBpedia (Zhang et al. 2015) which differentiates entity types by the beginning of their Wikipedia articles; CFPB-) which classifies consumer complaints into types; 20 newsgroups (Lang

---

Under review as a conference paper at ICLR 2021
1993
which includes short documents from 20 Usenet discussion groups; AG's News (hhang et al.,
2013 which contains a classification of news articles; ISEAR (Shao et al. 2013). which considers
personal reports for emotion; SMS spam (Almeida et al., 201T), which identifies spam messages, Po-
larity (Pang & Lee 2005). which includes sentiment analysis on movie reviews; Subjectivity (Pang
& Lee| 2004), which categorizes movie snippets as subjective objective; and CoLAA (Warstadt
ct al. 2018), which annotates sentences for grammatical acceptability;
Each dataset was split into train, dev, and test sets, keeping the original split, if exists, and otherwise applying 70% 111 split, respectively. Since we are interested in limited labeled data set- tings, we used the dev sets only for optimizing the baseline algorithms, denoted by Path-1 and Path-2 in Fig. []] The dev sets were nor used in the algorithmic flows denoted by Path-3 and Path-4 in that figure. Since MLM inter-training is computationally demanding, for DBpedia, AG's News, Yahoo! Answers and CFPB, which are relatively large datasets, we limit the sizes of the train/dev/test sets to 15K/3K/3KK instances respectively, by randomly sampling from each set") Links to all datasets are provided in Appendix
3.2 EXPERIMENTAL SETUP
In our main set of experiments we compare the performance of fine-tuning BERT-based models over a target task, for different settings of intermediate training. We consider four BERT-based settings, as described in Section and in Figure I Two baselines- (i) BERT, where no intermediate training applied, and (ii) BERT IEMLM. where MLM used as the inter-training task; and two settings that rely on clustering (1) BERTITCIIST where predicting cluster labels is used for inter-training, and (2) BERTTTMM.-.LLST. which combines the two intermediate tasks. In addition, we include several non-BERT baseline settings which rely on simpler models.
Training samples: For each setting, the final model training for the target task (fine-tuning, in the case of BERT-based models) is performed, per dataset, for training budgets varying between 64 and 1024 labeled examples. For each data size x, the experiment is repeated 5 times; each repetition representing a different sampling of x labeled examples from the train set. Sampling of training examples is matched between all settings. That is, for given dataset and train size the final training run for all settings is done with respect to the same 5 samples of labeled examples Results per repetition appear in Fig. 4 of Appendix B
Inter-training: Intermediate training, when done, was performed over the full train set for each dataset (ignoring instances' labels). We studied two implementations for the clustering task: k- means (Lloyd] 1982) and sequential Information Bottleneck (sIB) which is known to obtain better results in practice (Slonim al., (002) and in theory (Slonim et al. 2013). Based on initial ex- periments, and previous insights from works in the computer vision domain (Yan et al.) 2020) we opted for a relatively large number of clusters, and rather than optimizing the number of clusters per dataset, set it to 50 for all c The k-means baseline was run over GloVe (Pennington et al., 2014) representations following word stemming. We used publicly available implementation of SIB with its default configuration (i.e., 10 restarts and a maximum of 15 iterations for each single run). For the clustering, we used Bag of Words (BOW) representations on a stemmed text with the default vocabulary size (which is defined as the 10k most frequent words in the dataset). Our results indicate that inter-training with respect to sIB clusters consistently led to better results in the final performance the target task, compared to inter-training with respect the clusters obtained with k-means. Hence, in the main paper we present results using the sIB implementation only, while results including k-means discussed in Appendix C We also considered inter-training only on
3 We verified that relying on this limited set has no significant impact on the performance of BERTIMMIMM and BERTICcLUS compared to using the full dataset. The results are omitted for brevity.
*Note for reviewers: 11 of the total 400 samplings did not cover all target classes, and fine-tuning not performed, hence results reflect an average over less than repetitions. For 20 newsgroups, budget 64 all samples did not cover all classes and hence this data point is not presented. We plan to re-sample for completing the missing data points.
*Setting the numher of clusters be cqual to the number of classes resulted in inferior accuracy. In addition, we note that in practice one may not know how many classes truly exist in the data, so this parameter is not necessarily known in real-world applications. httpps///gihuu..oo/BBM/M.M
4

---

Under review as a conference paper at ICLR 2021
representative examples of clustering results, filtering given amount of outlier examples. Filtering did not improve results (data not shown).
Note that the run time of the clustering algorithms is negligible, and only takes a few seconds. The run time of the fine-tuning step the inter-training task takes five and a half minutes for the largest train set (15k instances) Tesla 1O0-PCIE-16GB GPU.
BERT byper-parameters: The starting point of the various inter-training settings is the BERT BASE model (1 10M parameters). BERT inter-training and fine-tuning were all performed using the Adam optimizer (Kingma & Ba 2015) with standard setting consisting of a learning rate 3 10-3, batch size 64, and maximal sequence length 128.
The number of training epochs varies by setting. For the baselines of BERT and ITRM fine- tuning was performed over 15 epochs, selecting the best epoch based on accuracy over the dev set. Since in a practical limited annotations budget setting one cannot assume that a labeled dev set is available, in our BERTIT:CLLST and TEMMLLLS settings we did not use a dev set, and fine- tuning was arbitrarily set to be over 10 epochs. For inter-training over the clustering results we used a single epoch for two reasons. First, loosely speaking, additional training over the clusters may drift the model too far towards learning the partition into clusters, which is an auxiliary task in a practitioner, single epoch
context, and not the real target task. Second, from the perspective training is preferred since it is the least demanding in terms of run time. For MLM inter-training used 30 epochs with a replication rate of We note that using fewer epochs yielded similar results. As this may be of interest for other researchers, provide the results in Appendix D
Non-BERTT baselines: As additional baselines, the same training samples were also used to train multinomial Naive Bayes (NB) and linear Support Vector Machine (SVM) classifiers, using either Bag of Words (BOW) GloVe (Pennington et al., 2014) representations. For GloVe, text is represented as the average GloVe embeddings of its tokens. This yielded four additional baseline settings: NBBow, NBGioVe, SVMBow and SVMGlove
4 RESULTS
Figure depicts the classification accuracy for the different experimental settings in each of the datasets, for varying labeling budgets. The SVMG baseline outperformed the other baseline settings, therefore, we only include it in this figure; Full results for all NB and SVM settings can be found in Appendix [
Dataset
BERT accuracy
BERTHOLST accuracy
Gain
Eror reduction
Yahoo!
24.7
86%
28%
Answers
DBpedia
42.2
67.0
59%
43%
CFPB
22.1
27.5
24%
7%
20 newsgroup
19.7
47.2
139%
34%
AG's News
73.3
80.7
10%
28%
all 6 topical datasets, BERTTCLUST and BERT clearly outperform BERT and BERTITMMM in the small labeled data regime, where the gain is most prominent for the smallest labeled data examined when only 64 labeled examples are available and gradually diminishes as more labeled samples are added. In the remaining 4 non-topical datasets the clustering inter- training does not appear to confer a similar benefit. Nevertheless, even in these datasets, IERTIT results are typically comparable to the baseline algorithms.
Evidently,
ISEAR avg. topical
19.0
29.0
52%
12%
62%
25%
SMS spam
98.0
98.2
0%
10%
Subjectivity
90.7
91.0
0%
3%
Polarity CoLA
67.7
67.0
-1%
-2%
69.5
66.0
-5%
-11%
avg. non-topical
-1%
0%
Table 2: Classification accuracy for BERT and BERT when using 64 samples for fine- tuning, the accuracy gain relative to BERT's ac- curacy and the reduction in error (1-accuracy) rel- ative to the BERT error.
Table depicts the results from Figure fo- cusing the practical use case of minimal budget of 64 samples for fine-tuning, As is evident, BERTI:CLUSE confers a significant benefit in accuracy (62% accuracy gain, 25% error reduction on average), when considering the topical datasets.
5

---

Under review as a conference paper at ICLR 2021
Yahoor Answers
D9pedo
50
100
90
80
50
Fccuncy
79
BERT
BERTy
nTRT
BEATin
OERT,
DERT
64
178
189
256
TMC.
517
764
1874
64
178
197
716
517
700
1074
fTae-buriing sampois
*fine-ttniing somples
20 resigroup
BERT
50
70
DETT
49
40
20
BEAT.
10
128
102
912
768
1024
125
256
394
512
765
1024
Hine tuning somples
Hin turing samples
DERT
67.5
GERT.
60
30
27.5
DEAT
30
125
DERT,
DERT
29
DERT
67.5
19
120
192 250
512
765
2024
128
192
236
304
1024
samples
CMS spam
Sutuectty by
100
96
94
42
DEAT
ZERT
BERT
-00
TeAT cuer
BTRT
84
64
17B
199
256
512
760
1874
64
17A
147
756
704
512
740
1674
fiae-huning camples
fnne-tuiing somplex
Polarity
COLA
80
DERT
BEST
DEX
77.5
BEET
CERT
BERT
BERT,
72.5
70
65
60
.in tuning somples
Hin turling samples
Figure 2: Classification accuracy over the test set for the different experimental settings versus the number of labeled samples used for fine-tuning (log scale). Each point is the average of five repetitions.
6

---

Under review as a conference paper at ICLR 2021
Train
64
128
192
256
384-512
768
1024
BERTICLUST
2x 1
3x +
2x 10
vs. BERT
BERTHCUUST'
2x10-4
2x 10
8 8x10
2x 10
vs. BERTIMMM
6 6x
1 1x
Table 3: Paired I-test
p-values (after Bonferroni correction) of classification accuracy for BERTIraCLse compared to BERT (top row) and to I (bottom row). Symbols indicate the setting which achieved better performance. denotes insignificant results (p 2 0.05).
Table 3 includes statistical significance analysis. We perform paired t-tests to compare IECLUST with BERT and BERT M- pooling together all datasets and repetitions for given labeling bud- get. For a budget up to 192 examples, BERTITCLLSTT significantly outperforms both BERT and BERTTMMMM expressing the robustness of the proposed approach.
BERT IT:CLUST BERTTTL also outperforms the non-BERT baselines. For a few topical datasets, SVMgiove on par with slightly superior to BERTTCCUUST for the smallest train budget. Overall, when considering the entire range of fine-tuning sample sizes, its results are inferior to our approach and it often performs quite poorly.
The different behavior between the topical datasets versus the non-topical datasets highlights that the CLUST configuration we considered seems most valuable for topical tasks. Specifically, we note that both clustering algorithms we examined rely BOW representations and correspondingly are better suited for topical data in finding a partition that reasonably approximates the true hidden partition of the data, according to the class labels. Therefore, we try using the BERT [CLS] token representation as a non-BOW representation. While seems somewhat better than BOW on the non-topical Polarity dataset, it is not consistently better on the other non-topical datasets, and it is mostly worse on the topical datasets (see Appendix C Future work should examine ways to make BERT better suited for non-topical datasets, possibly by capturing stylistic distinctions.
The performance gains of mERUTT over BERTTTMMM suggest that the potential benefits of BERTILLUST do not consist merely of adapting the model to the characteristics of the target class corpus; rather, it appears that inter-training on top of the clustering results carries additional benefit. A natural explanation that the pseudo-labels obtained via the clustering partition are informative with regards to target task labels. To quantify this intuition, in Figure 3 we depict the Normalized Mutual Information (NMI) between cluster labels and the target task labels, calculated over the entire training set, versus the gain of using BERT reflected as the reduction in classification error rate between BERT and I..UST at the extreme case of 64 fine-tuning fine-tuning samples. Evidently, in datasets where the NMI is around 0, BERTI does not confer clear benefit; conversely, where the NMI is relatively high, the performance gains are pronounced as well.
Finally, since the partition obtained via unsupervised clustering is often informative for the target class labels, we examine whether it can utilized directly, as opposed to as pseudo-labels for BERT inter-training. To that end, we applied a simple heuristic. Given labeling budget, we divide it across all clusters, and use the budget per cluster to reveal the labels a random sample of examples in that cluster, and then identify each cluster with the most dominant label found in it. Next, given a new test example, we assign it with the label associated with its nearest cluster. In Appendix B and Figure 4 we provide full details, and share the performance of this simple baseline. While such a rudimentary classifier can be surprisingly effective, especially where the NMI is high and the labeling budget is low, it is generally not on par with BERTTTCLUST performance.
Taken together, our analyses suggest that in topical datasets, where labeled data is scarce, the pseudo- labels generated via simple text clustering techniques can be leveraged by BERT for inter-training, to provide a better starting point of the model towards its final fine-tuning for the target task.
5 RELATED WORK
In our work, we transfer the pretrained BERT (Devlin et al. 2018) model to a new domain with little data. There is whole field studying how to transfer models across domains, namely, transfer

---

Under review as a conference paper at ICLR 2021
Error Reduction Rate
Error resuction rate for 83RTinsuss vs. BERT
40
30
20
AG's News
10
ISARR
SMS spam
Polarity
CoLA
Subjectivity
Yahoo! Answers
-10
DBpodia
0.0
0.1
0.2
0.3 0.4
0.5
0.5
0.7
Normalized Mutual Information (NMI)
Figure 3: Improvement by TECUUT vs Normalized Mutual Information per dataset. x-axis: Normalized Mutual Information between the cluster labels and the class labels, calculated over the entire train set. y-axis: The reduction in classification error (percentage) of inter-training using BERT relative to inter-training, when using 64 samples in the fine-tuning phase.
learning. It suggests methods such pivoting (Ziser & Reichart 2018). weak supervision (Shnarch et al. 2018). and adversarial transfer (Cao et al. 2018).
In Computer Vision, pretrained models often learnt by image clustering tasks (Caron et al. 2018
In Natural Language Processing, however, clustering was mainly used for non-transfer scenarios.
Ball (2019) relies on pretrained embeddings cluster labeled and unlabeled data. Then, they fill the missing labels to augment the training data of any classifier. Clustering itself was improved by
combining small amounts of data (Torres& Vaca 2019 Wang et 2016).
Pretrained models improved state-of-the fif-the-rt art in many tasks (Nouueira & Cho 2019 Ein Dor et al, 2020) and they are especially needed and useful in low resource and limited labeled data settings [Lacroix et al. 2019 Wang et al.] 2020aa Chau et al. 2020). There are many suggestions to improve such models, including larger models (Raffel et al.J 2019), changes in the pretraining tasks and architecture (Yang et al. 2211)) augmenting pretraining (Geva et al. (0200) or improving the transfer itself (Valipour et al. 2019 Wang et al. 2019b; Sun et al. 2019 Xu et al. 2020). Two findings on pretraining support hypothesis the intermediate task. namely, classification surpass MLM.
Some pretraining tasks are better than others (Lan et al.) 2020 Raffel et al. 2019) and supervised
classification additional pre-training improves performance (Lv et al. 2020 Wang et al.) [201aa] Pruksachatkun et al., 2020). All these works aim to improve the performance upon transfer, making more suitable for any new domain. In contrast, we focus on the improvement given the domain.
With transferred model, one can further improve performance with domain-specific information. For example, utilizing metadata (Melamud et al. 2019), training on weakly-supervised data (Raisi) & Huang 2018) or multitasking on related tasks concurrently (Liu et al. 2019a). Given no domain- specific information, was suggested further pretrain unlabeled data from the domain (Whang
et al. 2019 Xu et al. 2019 Sung et al.) 2019 Rietzler et al. 2020 Lee et al. 2020 Gururangan al. 2020p. This, however, sometimes unhelpful or even hurts results (Pan 2019). We replicate
this finding for cases in which labeled data is scarce (see Section (4)
Transferring a model and retraining with paucity labels often termed few shot learning. Few shot learning is used for many language related tasks such as named entity recognition (Wang al.) 2020b), relation classification (Hui et al.) 2020) and parsing (Schuster et al. 2019). There have also been suggestions other than fine-tuning the model. Koch (2015) suggests to rank the similarity between examples with Siamese networks. Vinyals et al. (2016) rely on memory and attention to find neighboring examples and Snell al. (2017) search for prototypes to compare to. Ravi & Larochelle (2017) don't t define in advance how to compare the examples. Instead, they meta-learn how to train the few shot learner. These works addressed the image classification domain, but they
8

---

Under review as a conference paper at ICLR 2021
supply general methods which are used, improved and adapted on language domains (Geng al. 2019 Yu et al. 2018).
In conclusion, separate successful practices foreshadow our findings: Clustering drives pre-training on images; supervised classification aids pre- pre-training; and training on unlabeled domain examples is helpful with ML..
6 DISCUSSION
We presented simple approach to improve BERT-based models for topical text classification. Specifically, we show that inter-training BERT over pseudo-labels generated via unsupervised sIB clustering results in model that represents better starting point for the final fine-tuning over the target task at hand. Thus, our analysis suggests that BERT can leverage these pseudo-labels, namely that there exists a beneficial interplay between the proposed inter-training and the fine-tuning stage. Our results show that this approach yields a consistent significant boost in BERT accuracy over topical data when labeled data is scarce.
We opted here for a practically oriented approach, which we do not claim to be optimal. Rather, the success of this approach suggests various directions for future work. In particular, several theoretical questions arise, such as what determines the success of the approach in a given dataset; understand the potential synergistic effect of using BOW-based clustering for inter-training BERT representa- tions; could more suitable partitions be acquired by exploiting additional embedding space and/or more clustering techniques; co- co-training (Blum & Mitchell 1998) methods, and more.
On the practical side, while in this work we focused on inter- training over 50 clusters for a single epoch, more work is needed to determine how to tune such syper-parameters In addition, one may consider using the labeled data available for fine-tuning anchors for the intermediate clustering step, which we have not explored here.
Another point to consider is the nature of the inter-training task. Here, we examined multi-class setup where BERT trained to predict one out of nhe cluster labels. Alternatively, one may consider binary inter-training task, where BERT is trained to determine whether two samples are drawn from the same cluster or not.
Finally, the focus of the present work In principle, inter- trairig BERT over clustering results may be valuable for additional downstream target tasks, that are similar in spirit to standard text classification. Examples include recent work Key-Point Analysis (Bar-Haim) et al.) (200)) and the task of Textual Entailment (Dagan et al. 2013). The potential value of our approach these and other cases left for future work.
improving BERT performance for text classification.
REFERENCES
Tiago A. Almeida, José María Gómez Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: new collection and results. In Matthew R. B. Hardy and Frank Wm. Tompa (eds.), ACM Symposium Document Engineering, pp. 259-262 ACM, 2011. ISBN 978-1- 4503-0863-2. URL httphtttp///lppp uni teilee/.d//bb/ccof////e///////// html #Almeidam1ii
M. Ball. Brown university ripped Recursive intent propagation using pretrained embedding dis- tances. 2019.
Roy Bar-Haim, Lilach Eden, Roni Friedman, Yoav Kantor, Dan Lahay, and Noam Slonim, From arguments to key points: Towards automatic argument summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), 2020. URL https:
/ www.. wwwebb org/ http///nntooooy///.ooooooo acl main. 371. .df
Avrim Blum and Tom Mitchell. Combining labeled and unlabeled data with co-training. In Pro- ceedings of the Eleventh Annual Conference Computational Learning Theory, COLT" 98, pp. 92-100, New York, NY, USA, 1998, ACM. ISBN 1-58113-05770, doi: 10.1145/2799433279992.
URL http: //do am.. 0ttg:/....4//2799433.11

---

Under review as a conference paper at ICLR 2021
Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao, and Shengping Liu. Adversarial transfer learning for chinese named entity recognition with self-attention mechanism. In EMNLP, 2018.
M. Caron, P. Bojanowski, Armand Joulin, and M. Douze. Deep clustering for unsupervised learning of visual features. In ECCV, 2018.
Ethan C. Chau, Lucy H. Lin, and Noah A. Smith. Parsing with multilingual bert, a small corpus, and a small treebank. ArXiv, abs/2009. 14124, 2020.
Ido Dagan, Dan Roth, Mark Sammons, and Fabio Massimo Zanzotto. Recognizing textual en- tailment: Models and applications. Synthesis Lectures on Human Language Technologies, 6(4): 1-220, 2013.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2018
Liat Ein-Dor, Eyal Shnarch, Lena Dankin, Alon Halfon, Benjamin Sznajder, Ariel Gera, Carlos Alzate, Martin Gleize, Leshem Choshen, Yufang Hou, et al. Corpus wide argument mining-a working solution. In AAAI, pp. 7683-7691, 2020.
Ruiying Geng, Binhua Li, Yongbin Li, Xiao-Dan Zhu, Ping Jian, and Jian Sun. Induction networks for few-shot text classification. In EMNLP/IJCNLP. 20119 2019.
Mor Geva, Ankit Gupta, and Jonathan Berant. Injecting numerical reasoning skills into language models. In ACL, 2020.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. ArXiv, abs/1803.07728, 2018.
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't 't stop pretraining: Adapt language models to domains and tasks. ArXiv, abs/2004. 10964, 2020.
Bei Hui, Liang Liu, J. Chen, X. Zhou, and Yuhui Nian. Few-shot relation classification by context attention-based prototypical networks with bert. EURASIP Journal Wireless Communications and Networking, 2020:1-17, 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, /bbs/142,.690,, 2015.
Gregory R. Koch. Siamese neural networks for one-shot image recognition. 2015.
A. Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual representation learning. 2019 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1920-1928 2019.
O. Lacroix, Simon Flachs, and Anders Søgaard. Noisy channel for low resource grammatical error correction. In BEA ACL, 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. In International Conference on Learning Representations, 2020. URLhtttss: hhtttss://peerreview ww..ne/ffrrri
lid cd-HeeA7AEvs)
Ken Lang. Newsweeder:
Learning to filter netnews. In Proceedings of the Twelfth International Conference on Machine Learning, pp. 331-339, 1995.
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, D. Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang- Biobert: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 2020.
Xiaodong Liu, Pengeheng He, W. Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. ArXiv, abs/1901 11504, 2019a.
10

---

Under review as a conference paper at ICLR 2021
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, 2019b.
Stuart Lloyd. Least squares quantization in pem. IEEE transactions on information theory, 28(2): 129-137, 1982.
Shangwen Lv, Yuechen Wang, Daya Guo, Duyu Tang, N. Duan, F. Zhu, Ming Gong, Linjun Shou, Ryan Ma, Daxin Jiang, G. Cao, M. Zhou, and Songlin Hu. Pre-training text representations as meta learning. ArXiv, abs/2000.0556.. 2020.
Oren Melamud, Mihaela A. Borna,, and Ken Barker. Combining unsupervised pre-training and annotator rationales to improve low-shot text classification. In EMNLP/UJCNLP, 2019.
Rodrigo Nogueira and Kyunghyun Cho. arXiv:1901.04085, 2019.
Passage re-ranking with bert.
arXiv preprint
Chenchen Pan. Analyzing bert with pre-train on squad 2. O. In Stanford Archive, 2019.
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based minimum cuts. In Proceedings of the 42nd Annual Meeting of the Associa- tion for Computational Linguistics (ACL-04), pp. 271-278, Barcelona, Spain, July 2004. doi: 10.3115/1218555.1 12111228955.12119999 1218990. URLhtttpe /ttp:://wwwww... www.ww.wwww......... org/ wttp://///...ooooo/o-o-----
Bo Pang and Lillian Lec. Sceing stars: Exploiting class relationships for sentiment categoriza- tion with respect to rating scales. In Proceedings of the 43rd Annual Meeting of the Associa- fion for Computational Linguisties (ACL'05), pp. 115-124, Ann Arbor, Michigan, June 2005.
Association for Computational Linguistics. doi: 00.3115/121944011199555 URL https:
7/www. www.:aawwwwbb.rro org htt.a//wwww.or.aaaaoooooo///----
Jeffrey Pennington, R. Socher, and Christopher D. Manning. Glove: Global vectors for word repre- sentation. EMNLP, 2014.
Jason Phang. Thibault Févry, and Samuel R Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv: 18/ 1.00888, 2018.
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R Bowman. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work? arXiv, pp. arXiv- 2005, 2020.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/1910.10682 2019.
Elaheh Raisi and Bert Huang. Weakly supervised cyberbullying detection with participant- vocabulary consistency. Social Nenwork Analysis and Mining, 8:1-17, 2018.
Ravi and H. Larochelle. Optimization as a model for few-shot learning. ICLR, 2017.
A. Rietzler, Sebastian Stabinger, Paul Opitz, and Stefan Engl. Adapt or get left behind: Do- main adaptation through bert language model finetuning for aspect-target sentiment classification. ArXiv, abs/1908. 1860, 2020.
Tal Schuster, Ori Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of contextual word embeddings, with applications to zero-shot dependency parsing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 1599-1613, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 10.18653/v1/N19-1111 1162. URL https: http://wwwwwwwwll.cc
ac web. org/ httpee/..rgggaatooooooooooo
Bo Shao, Lorna Doucet, and David R. Caruso. Universality versus cultural specificity of three emo- tion domains: Some evidence based the cascading model of emotional intelligence. Journal of Croxs Cuitural Psychology, 46(2):229-25 251,, 2015. URL hctps: :////. orq/10.1177/
00220021144577479

---

Under review as a conference paper at ICLR 2021
Eyal Shnarch, Carlos Alzate, Lena Dankin, Martin Gleize, Yufang Hou, Leshem Choshen, Ranit Aharonov, and Noam Slonim. Will it blend? blending weak and strong labeled data in a neural network for argumentation mining. In Proceedings of the 56th Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 1pp.599-600 599-605, Melbourne, Australia, July 2018. Association for Computational Linguisties. doi: 10.1186//11/1812000 URL https:
77 www. all.aclweee www.:orgggaatnn.ee http//anhhoooyyyyooe 2095
Noam Slonim, Nir Friedman, and Naftali Tishby. Unsupervised document classification ing sequential information maximization. In Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development Information Retrieval, SIGIR '02, pp. 129-136, 2002. doi: 10.1 10.1145/564376.564401. URL https://dcc. org 10.154/
564376. 564401
Noam Slonim, Ehud Aharoni, and Koby Crammer. Hartigan's k-means vs. lloyd's k-means is time for a change? In Proceedings of the 23rd International Joint Conference on Artificial Intelligence (IJCAI), 2013. doi: 00.5555/2540128.5444440 URL httpt::t/tl.aaaooe org/
2540366
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in neural information processing systems, pp. 4077-4087, 2017.
C. Sun, Xipeng Qiu, Yige Xu, and X. Huang. How to fine-tune bert for text classification? ArXiv, abs/1905, 005033,.555..... 2019.
Chul Sung, Tejas I. Dhamecha, Swarnadeep Saha, T. Ma. V. Reddy, and R. Arora. Pre-training bert on domain resources for short answer grading. In EMNLP/JJCNLA 2019.
Johnny Torres and Carmen Vaca. Cl-aff deep semisupervised clustering. In AffCon@ AAAI, 2019.
Mehrdad Valipour, En-Shiun Annie Lee, Jaime R. Jamacaro, and Carolina Bessega. Unsupervised transfer learning via bert neuron selection. ArXiv, abs/1912.053008 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. In Advances in neural information processing systems, pp. 3630-3638, 2016.
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Na- joung Kim, Ian Tenney, Yinghui Huang. Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computationnll Linguistics, pp. 4465-4476 Flo- rence, Italy, July 2019a. Association for Computational Linguistics. doi: 10. 10551/944//999994443 URL https: www aclweb. org/ http///aahooooooooeaeaeo
R. Wang, Haibo Su, Chunye Wang, Kailin Ji, and J. Ding. To tune or not to tune? how about the best of both worlds? ArXiv, abs/1907, 0053380,.. 2019b.
Sinong Wang, Madian Khabsa, and Hao Ma. To pretrain or not to pretrain: Examining the benefits of pretraining on resource rich tasks. In ACI., 2020a.
Yaqing Wang, Subhabrata Mukherjee, H. Chu, Yuancheng Tu, Miaonan Wu, Jing Gao, and Ahmed Hassan Awadallah. Adaptive self-training for few-shot neural sequence labeling. ArXiv, abs/2010.03680, 2020b.
Zhiguo Wang, Haitao Mi, and Abraham Ittycheriah. Semi-supervised clustering for short text via deep representation learning. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pp. 31-39, 2016.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 1805.
10.800/.....11 /247/ 2018.
T. Whang, Dongyub Lee, Chanhee Lee, Kisu Yang, Dongsuk Oh, and Heuiseok Lim. Domain adaptive training bert for response selection. ArXiv, /0//1900004881 2019.
12

---

Under review as a conference paper at ICLR 2021
Hu Xu, Bing Liu, Lei Shu, and Philip Yu. BERT Post- Training for Review Reading Compre- hension and espect-based Sentiment Analysis. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 2324-2335, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/1//N19-12422 URL
www.waclweb.ooo htwp://wwwwb...rr///t.oooooo////
Yige Xu, Xipeng Qiu, L. Zhou, and X. Huang. Improving bert fine-tuning via self-ensemble and self-distillation. ArXiv, abs/2002. 10345, 2020.
Xueting Yan, I. Misra, Abhinav Gupta, Deepti Ghadiyaram, and D. Mahajan. Clusterfit: Improving generalization of visual representations. 2020 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 6508-6517, 2020.
Z. Yang, Zihang Dai, Yiming Yang. J. Carbonell, R. Salakhutdinov, and Quoe V. Le. Xlnet: Gener- alized autoregressive pretraining for language understanding. In NeurIPS, 2019.
Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. Diverse few-shot text classification with multiple metrics. In Pro- ceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long Papers), pp. 1206- 1215, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v 109. URL https: www.aclwwwt...........w.w.wwwwww http://cwwwb..........oooooo////
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. In Advances in neural information processing systems, pp. 649-657, 2015.
Yftah Ziser and Roi Reichart. Pivot based language modeling for improved neural domain adapta- tion. In NATCL-HLT, 2018.
A DATASETS
In this paper we used 10 datasets. Table T provides details about their split into train, dev, and test sets. For each set its size and the prior of the target class is presented.
Links for downloading the datasets:
httpi:/hhh..toooo/./cocoooooooo/ movie-reviewwdatall
CFPB:
http::///ww....oooer/rraaaaaooos consuumer--commpaannsee
20 newsgroups: htt.s///sssssoo///oooooo/o//////
We used the version provided by scikit: 1ttp://///cXttt-lmmooo//...33
@atasets/twenty http:/entttyyessrruussustt httml html
AG's News, DBpedia and Yahoo! answers: We used the version from: https://pathmind,
com./wikii/ppee-dattteoo
(look for the link Text Classification Datiiii Datasets).
SMS spam:
ISEAR:
research-materiall
CoLA:
B SIB-BASED CLASSIFIER
As described in section 4 experimented with building rudimentary classifier that utilizes only the sIB clustering results and the labeling budget. We estimate the most common label for each cluster by labeling some of its instances, using given labeling budget. The budget is distributed
13

---

Under review as a conference paper at ICLR 2021
among the clusters relative their size, while ensuring that at least one instance from each of the 50 clusters labeled. Then, each cluster is assigned the majority label from its labeled instances. Test examples classified according to the label associated with the nearest cluster. Results for this setting are depicted Fig. 4 Comparing these results to the BERT-based approaches reveals that clustering alone is not sufficient.
C EXAMINING ADDITIONAL CLUSTERING TECHNIQUES
In addition to sIB over BOW (denoted BERT ICLLSST) evaluated three more configurations for the clustering intermediate task; K-means over GloVe representation, K-means over BERT CLS, and Hartigan's K-means (Slonim al. 2013) over GloVe. For BERT CLS, for each input text take the representation of the [CLS] token from the last hidden layer of the BERT model.
Results are shown in Fig. When comparing sIB, on eight of ten occasions sIB over BOW outperforms the other clustering configurations.
In initial trials, sIB was also showing better results as a clustering method, which the reason we used it rather than K-means as our main intermediate tasks. Because of the change in quality, there is not enough evidence to say whether sIB is better as pretraining task due clusters that are more related to the target task or due to some characteristics of the clusters themselves. Such different characteristics may arise, for example, from the ways clusters are chosen. K-means relies on Geometric notions such as distance to cluster while sIB on information theoretic notions.
D THE EFFECT OF THE NUMBER OF MLM EPOCHS
We tested several options for the number of epochs of intertraining using MI... The results in Fig. don't t show clear choice, although all cases are not consistently improving over the baseline. We thus chose 30 epochs, MLM scores seemed somewhat more favourable.
E NON-BERT BASELINES
The results of the NBBow, NBGlove. SVMBow and SVMG baselines are shown in Figure 7
14

---

Under review as a conference paper at ICLR 2021
Yahoor Answers
D0peda
50
100
80
50
20
64
178
189
256
517
764
1874
64
17A
197
716
512
760
1074
fiae-buriing sampiis
#fnne-tuninng somplex
20 resigroup
FERT
BERT
70
DE
+
40
SERT
BERT cuar
20
BEST ua GUIT
10
128
102
512
768
1024
256
384
1024
tuning somples
Hine turinng samples
DERT
BERT.
60
Accuney
40
BEAT
30
BERT
29
BENI
19
120
192
250
512
765
2024
128
192
236
304
1024
GMS spam
Saltectty
100
100
96
80
75
65
50
40
64
17B
192
254
512
760
1074
64
12A
197
756
412
740
1674
frae-turing camples
fine-tuiing somples
Polority
COLA
80
70
66
"EER
35
50
Pin tuning somples
Hin turling samples
Figure 4: Classification accuracy over the test set for the different experimental settings versus the number of labeled samples used for fine-tuning. Each point in the line is the average of five repetitions. The repetitions themselves are shown. The results of the slB-hased classifier are also shown. X axis denotes the budget for training in 15g scale, and Y accuracy of each model.

---

Under review as a conference paper at ICLR 2021
Yahoot Anseerss
D0pedo
70
100
90
20
70
PERT
RFRT
SEIT'P
DEAT
40
SENT
64
17B
180
256
517
764
1874
64
178
197
716
512
700
1074
Tine-tuning samppis
Fine-tuniing samples
20 ressgrru
70
Accurny
128
192
765
1024
256
1024
tuning mplls
Hine turiing samples
67.5
CERT
60
30
40
27.5
BAKI
125
DEAT
67.5
19
120
192
250
512
765
2024
192
236
304
1024
samples
GMS spam
Suttecctv
59.6
55
99.4
20.2
BEIfT At-
94
BERT. A-eamiisi
ACCUIOC
-3
58.6
58.2
35.4
SERT
BEATICE
DEAT
91
90
64
17B
192 254
512
1074
64
17A
147
756
304
412
74A
1674
Tine-huning samples
Fine-tiiing somples
Polerity
COLA
87.5
82.5
74
pocurac)
77.5
72
72.5
66
67.5
64
2014
62.5
60
+in cuning somples
HinC Durling samples
Figure 5: Comparing various clustering configurations: (1) K-means over Glove, (2) K-means over CLS. (3) Hartigan's K-means over Glove, (4) sIB over BOW, and (5) no intermediate task. Each point is the average of five repetitions. X axis denotes the budget for training in log scale, and Y 16
accuracy of each model.

---

Under review as a conference paper at ICLR 2021
Answers
Copedo
07
BEMT an 30 spocts BCRT
1.0
0.6
S-caDers BERT..
0.9
0.5
0.5
ACCUIKK
0.4
BENT
0.4
upoch BoRl
cpodis Mofti
0.3
BEAT
64
178
180
256
3EC
912
768
1074
64
178
197
716
512
760 1074
Eineduuing tamples
Fiseduiing samples
20 ruuiggouu
MRRT
opocc SERTT
Lopect BENT
DETIc
0.45
aaxlo BoMty
0.59
0.25
768
Hin Buriing samples
Hhe turling samaless
0.875
0.6
0.850
opochs BEfITy acM
0.825
0.5
0.000
0.775
0.750
03
epoch
0.725
apcche and
0.700
epoche BERT
82
120
192
765
2024
236
GVS spam
Suttectty
opach BERTy ugr
30 opocts SERT,
0.58
opachs BERT
ACCrOC
0.56
BERT
091
spach EEIT,
enclls BbH
17B
182
512
36
1074
64
128
197
316
304
912
740
1674
Ennettriin sumples
fheddiingg cemples
Potarity
COLA
BEAT
rpor BERTe
oETTs
0.80
0.76
0.73
0.70
072
#pach BEWTs spocts BEIT, aur
#nC turing samples
Hio turiing samoles
Figure 6: Evaluating the effect of different number of MLM epochs. Each point in the line is the average of five repetitions. X axis denotes the budget for training log scale, and Y accuracy of each model.
17

---

Under review as a conference paper at ICLR 2021
Yahoor Answers
Depedao
70
40
Fecuncy
40
BENT
50%
SeNaww
20
9/Mose
50
64
178
189
256
517
764
1874
64
178
197
216
512
760 1074
fiae-hutingg sampois
#fnne-tuningg samples
20 ressgrou
BE
CERT.
70
30
128
192
512
768
1024
125
256
394
1024
tuning somples
Hin turing samples
BERT
60
Mase
Aecuney
20
40
30
509
29
19
120
192
250
512
765
2024
128
192
236
304
1024
samples
GMS spam
Suttectty by
100
100
40
24
85
BEAT
SEnT:
Schlow
70
64
178
189
254
512
766
1974
64
17A
197
756
412
740 1674
(Tae-huning camples
fine-tuiing somplex
Polority
COLA
BEAT
BERT
75
Milao
65
59
.in buning somples
H.n turling samples
Figure 7: Comparing nOn-BERT baselines and the BERTI.LLSST setting. Each point is the average of five repetitions. X axis denotes the budget for training log scale, and Y accuracy of each model.
18


Reviews:
[{"review_id": "jbGR7Ji_JOF", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to use a simple intermediate task - clustering - to improve the generalization ability of BERT in low resource text classification settings. The idea is simple. We can perform clustering using BoW representations to generate pseudo labels, which will be used to fine-tune the pre-trained BERT model. The results show that the proposed method is effective in topical classification problems.\n\nPros:\n - the idea is very simple and we can easily adapt the idea to different classification tasks.\n- the results are promising, especially on topical classification problems.\n\ncons:\n- My main doubt about this paper is that I think the improvements in the topical classification problems were rooted in the fact that the authors used the \"BoW representations\" to conduct the clustering. The BoW representations are known to be more effective for topical classification problems compared to BERT-based sentence representations. So what the authors did maybe actually infuse/distill the BoW knowledge into the BERT representations.\n- I suggest the authors include a set of BoW representation based baselines. E.g., simple BoW based SVM model, average of GloVe embeddings of BoW representations based classification.", "extra_scores": {}, "extra_text": {}}, {"review_id": "WLR1hHoO3tz", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed an inter-training framework of BERT, an unsupervised training method applied between the conventional pre-training method via Masked Language Models and the fine-tuning method by using labeled data in the target task. The proposed inter-training methods are an unsupervised method by first running clustering with BoW features for the target task and then fine-tuning by training BERT over pseudo-labels generated by clustering results. This inter-training framework significantly improves prediction accuracies especially when the task is topical, i.e., the task to classify texts based on a high-level distinction related to what the text is about, and the labeled data is scarce.\n\nThis paper is well-written. The motivation is reasonable and the proposed methods make sense. Experiments are comprehensive and analyses are well designed. The contribution of this paper is obviously above the ICLR borderline.\n\nI think it is better if the authors mention the computational cost of the inter-training framework in detail. While the computational cost of the clustering is negligible, the authors did not mention the computational cost of the fine-tuning in the inter-training framework. Compared with the final fine-tuning process, it is better to clarify how much additional cost we need to perform the inter-training method. Moreover, it is better if the authors mention the effect of the size of unlabeled data in the inter-training method. Is more unlabeled data improve target metrics a lot, or are hundreds of texts sufficient for the inter-training method. It is better if the authors show the plot by changing the size of the unlabeled data used for the inter-training method.", "extra_scores": {}, "extra_text": {}}, {"review_id": "PwL1kwPLh8r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel domain/task adaptation procedure for BERT-style language models (LMs). Inspired by computer vision, the authors propose to specialize LMs to a particular domain and task with an intermediate clustering task. A large unlabeled dataset is clustered to generate pseudo-labels then a BERT model is specialized to predict the cluster id of each example. More precisely, the authors use GloVe embeddings (bag-of-word representations) and the sIB clustering algorithm with a constant number of clusters (50). \n\nThe additional clustering task is shown to help for low-resource topic classification. In the paper setup, the whole dataset (up to 15000 examples) is used for clustering (without the labels), while the fine-tuning phase relies on a small subset of the training data. The authors show nice gains when using 64 examples for fine-tuning. Graphs show that the gains are lower as more fine-tuning data is used, and usually disappear with more that 192 to 384 examples. The authors go on showing that their approach works best when the cluster ids correlate well with the original labels.\n\nOverall the paper is clear and well-written. It proposes an interesting and, to the best of my knowledge, novel idea.\n\nConcerns:\n- Some claims feel overly broad. I believe the claims should be limited to improving low-resource/few-shot *topic* classification tasks, not text classification in general, which is especially unclear in the title and abstract.\n- While the domain transfer literature is well introduced, no low-resource/few-shot literature is mentioned, making it hard to relate the results to similar research (although to be fair, it is true that CV has much more literature on this topic than NLP).\n- There is little discussion on the motivation and practicality of this approach, which makes the results seem almost anecdotic. If the motivation is to improve few-shot learning, the relevant literature should be introduced and compared against. If the motivation is topic classification, the practicality of this work seems limited: since a larger dataset is available for clustering, manually labeling a few hundred examples (human expert, MTurk) feels more promising. For example, BERT with 256 examples yields better results on most datasets than BERT:CLUST with 64 examples.\n- Several ideas mentioned as future work seem relatively straightforward to try and could have been interesting for inclusion in this paper.\n\nQuestions:\n- The correlation between cluster ids and task labels seems critical for the approach to work. Did the authors try using a number of clusters equal to the number of labels? If so, how did that change the results?\n- The vocabulary size used for clustering (10000 word stems) seems low. What are the reasons for choosing that number? Also, is the vocabulary task-dependent?\n- Why did the authors use a BoW approach for clustering? For example, why not embed each example with BERT (which should capture topics better than GloVe-BoW) using the output [CLS] vector?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "5AfAlFvToVR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality\n\nThe paper proposed using unsupervised clusters to help boost BERT performance in text classification tasks that lack labels. The method make good sense. Experiments well designed and showed clear advantage.\n\nClarity\n\nThe paper is well written. I have no trouble following all details.\n\nOriginality\n\nIt might be the first for BERT, but using unsupervised clustering to help classification seems an old topic in NLP. \n\nSignificance\n\nIt's less significant. First, it limits to topical classification. Second, it fits the scenario with more data for clustering (thus not truly low resource) but just too few labels. Performance gain diminishes quickly as number of labels get over a few hundreds.\n\nWhy not higher rating? The method seems too intuitive from NLP research. It's about BERT but the idea is not that novel. It's only helping topical cases with quick diminishing gains. \n\nWhy not reject? Paper is well written, evaluation done thoroughly and showed good improvement. Authors analyzed the result in a useful way.\n\nDetailed comments:\n1) The authors should be careful about the \"low resource\" claim, as we still need fair amount of data, unlabeled, for the clustering.\n2) Did you use training set for clustering, without considering their labels? Seems so but not clarified in the paper.\n3) Why use accuracy, not P/R as the metric? are all test sets balanced?\n4) For Figure 3, consider using shapes or in-figure labels. Color dots are hard to read when we have many labels, and not friendly to color-blind people either.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper suggests a simple variant for BERT training that improves classification for smaller training samples.  So it has a very specific applicability unlike other published variants which generally improve a broad range of tasks.  The variant adds a self-supervision classification task based on clustering.  Experiments are done but it only shows improvement for small training sizes.

AnonReviewer4 suggested a BOW experiment/baseline which was done by the authors in an updated version.  This confirmed the authors line.  AnonReviewer3 asked for computational details, which were added.  AnonReviewer1 lists a number of limitations which the authors need to address and rephrase the statements in their paper.

So it is publishable work, but somewhat marginal due to its specialised nature and thus rejected.
Decision: Reject