Few-Shot Text Classification with Dual Contrastive Consistency Training
Authors: Liwen Sun
Venue: Submitted to ICLR 2023 (2023)
Keywords: Few-Shot Learning
Contrastive Learning
Consistency Training

Abstract: In this paper, we explore how to utilize pre-trained language model to perform few-shot text classification where only a few annotated examples are given for each class. Since using traditional cross-entropy loss to fine-tune language model under this scenario causes serious overfitting and leads to sub-optimal generalization of model, we adopt supervised contrastive learning on few labeled data and consistency-regularization on vast unlabeled data. Moreover, we propose a novel contrastive consistency to further boost model performance and refine sentence representation. After conducting extensive experiments on four datasets, we demonstrate that our model (FTCC) can outperform state-of-the-art methods and has better robustness. 

Under review as a conference paper at ICLR 2023
FEW-SHOT TEXT CLASSIFICATION WITH DUAL CONTRASTIVE CONSISTENCY TRAINING
Anonymous authors
Paper under double-blind review
ABSTRACT
In this paper, we explore how to utilize pre-trained language model to perform few-shot text classification where only a few annotated examples are given for each class. Since using traditional cross-entropy loss to fine-tune language model under this scenario causes serious overfitting and leads to sub-optimal general- ization of model, adopt supervised contrastive learning few labeled data and sesstency--rguuliizatiio on vast unlabeled data. Moreover, we propose a novel contrastive consistency to further boost model performance and refine sen- tence representation. After conducting extensive experiments on four datasets, we demonstrate that model (FTCC) can outperform state-of-the-an methods and has better robustness.
INTRODUCTION
Text classification is fundamental task in natural language processing with various applications such as question answering (Rajpurkar et al. 2016), spam detection (Shahariar et al. (2099) and sen- timent analysis (Chong et al. 2014). With the advancement deep learning, fine-tuning pre-trained language movel(Devlin et al. 20199 Liu et al. 2019) achieves significant success. However, it still requires large amount of labeled data reach optimal generalization of model. Thus, researchers gradually focus on comi-supervised text classification where only few annotated data is provided. The success of superviee methods results from the usage of abundant unlabeled data: Unla- beled documents in training dataset provide natural consistency regularization by constraining the model predictions to invariant to small noises text input(Xis (ie et al. 2019: Miyato et al. 20166]
Chen et al. 2020a). Despite mitigating the annotation burden, these methods are highly unstable in different runs and can still easily overfit on the very limited labeled data.
Inspired by the success of supervised contrastive learning under few-shot settings Gunel et al. 2021 Chen et al., 2022), we hypothesize that the learned contrastive representation under this scenario can help tackle aforementioned high variance issue and impose additional constraints the model. Label information and feature structure can simultaneously be propagated from labeled examples to unlabeled ones. Thus, devise novel contrastive consistency schema to further boost model per- formance. To validate the effectivvness and robustness of FTCC, we conduct extensive experiments on four datasets. The result of our experiments confirms that FTCC can be leveraged to improve the performance of few-shot text classification.
Based on this motivation, the contributions of this paper are follows:
We integrate supervised contrastive learning objective into consistency-regularizee semi- supervised framework perform text classification under few-shot scenario.
We devised a novel contrastive consistency schema to propagate feature structure from labeled data unlabeled data dynamically.
We demonstrated our model's superiority over state-of-the-ans semi-supervised methods and analyze the contribution of each component of FTCC through ablation study and also visualize the learned instance representations, showing the necessity of each loss and the advantage of FTCC representation learning over BERT fine-tuning with cross-entropy.

---

Under review as a conference paper at ICLR 2023
2
RELATED WORK
2.1
SEMI-SUPERVISED TEXT CLASSIFICATION
Increasing attentions are paid to semi-supervised text classification, since abundant unlabeled data are easier to obtain compared to labeled data. For instance, MMiyato et (2016) adds perturbations to Instance,MMiyato
word embedding from bidirectional LSTM(Graves & Schmidhuber (205)) and employs adversarial and virtual adversarial training in the text domain. Xie et al. (2019) enforces consistency regu- larization between unlabeled data and augmented data after back-translation. [Chen et al. (2020a) introduces an interpolation-base augmentation and regularization technique to extend consistency training work and better leverage vast unlabeled data.
2.2 CONTRASTIVE LEARNING
Contrastive Learning recently achieves remarkable success in self-supervised representation learn- ing on computer vision (He et al. 2019 Chen et al. 2020b) and natural language processing (Fang & Xie 2020; Yan al. (021). The basic idea of unsupervised contrastive learning is that after generating different views of the sample example, the model adopts loss function that can pull an anchor and a positive view closer and push the anchor away from other negative examples in the em- bedding space. Its effectiveness on downstream linear classification is justified through alignment and uniformity (Wang Isola 2020 Arora et al.) 2019). Learning good generic representation is widely investigated. For example, in natural language field. Fang & Xie (2020); Wang et al. (2021) adopt data augmentations such as word substitution, back translation, word reordering to generate different views of samples. Gao et al. (2021b): Yan al. (2021) adjust dropout ratio generate pos- itive pairs. Subsequently, researchers propose supervised contrastive learning (Khosla et al., (2020) loss that enforces representations of examples from the same class to be similar and the ones from different classes to be distinct. Guunel et al. (2021) adopts this loss to fune-tune pre-trained language model and brings astonishing improvement in low-resource scenario. Chen al. (2022) introduces label-aware data augmentation and utilizes supervised contrastive learning to simultancously obtain discriminative feature representations of input examples and corresponding classifiers the same space.
3 METHODOLOGY
3.1 PROBLEM FORMULATION
The task of our model to train a text classifier that efficiently utilizes both annotated data and unannotated data in semi-supervised learning. We denote that D' {(x], us), 1...., ,.) as labeled training data and n { = .,..., m as unlabeled training data where n is the number of the labeled examples, is the number of the unlabeled examples and n
3.2 SUPERVISED CONTRASTIVE LEARNING
Unlike traditional fine-tuning pre-trained language model, we additionally includes a supervised contrastive learning term fully leverage the supervised signals (Gunel et al.) 2021 Khosla al.) 2020). This mechanism takes the samples from the same class as positive samples and the samples from different classes as negative samples. The contrastive loss and cross-entropy loss defined as follows:
1
Lee
C (po[vll))))
N
exp(x) 24y Tac)
Last
Σ j=1
1 i 1
log N -== liyk exp(x) xh/Tac)
N
1=1
where we work with batch of training examples of size N, x E Rd is la normalized embedding the [CLS] token from an encoder to represent an example. N,, is the total number of examples in
the batch that have the same label yi: Tact 0 is an adjustable scalar temperature; the symbol
2

---

Under review as a conference paper at ICLR 2023
denotes the dot product; y denotes the true label and po(ysla)) denotes the model classifier output for the probability of the lth example after softmax.
3.3 CONSISTENCY TRAINING
Back-translation Sugiyama Yoshinaga 2019 Edunov et al. 2018) is common data augmentation technique and can generate diverse paraphrases while preserving the semantics of the original sen- tences. We utilize back translations to paraphrase the unlabeled data to get augmented noisy data. Following Xie et al. (2019), we minimize the Kullback-Leible divergence between the augmented view of the example and the original view of example to encourage their consistency and enforce the smoothness of the model. FTCC adopts the loss function follows:
Leon = (RI. (oor://p"""""""oooooo("""""
where is back-translation transformation, Pa(y|=²) is original example's label distribution through classifier, po (=())a(x"))) is the back-translated example's 's label distribution through classifier. denotes the stop gradient of 0, as suggested by VAT (Miyato et al. 2017).
3.4 CONTRASTIVE CONSISTENCY
Inspired by Xie et al. (2019); Wei al. (021)) we conjecture that consistency regularization not only can propagate label information from label data to unlabeled data, but also can propagate feature structure in their latent space. Thus, we propose to encourage the consistency between the feature pattern of original examples and their corresponding augmented examples, which further refines deep clustering and classification accuracy on augmented data.
We randomly sample batch of N examples and perform back- trccksrranlttoon every example, result- ing in 2NV data points. Given one example and its corresponding back-translated example, we treat the other 2(N 1) examples negatives. Then denote the similarity between the example x" and the negatives n, E 2(N 1)})
in one batch as:
exp(x" nu/Tee)
P(i) 2(N exp(x" n2 ng/Tee)
Likewise, the similarity between the corresponding augmented example a(*") and the negatives ni(i E {l..... 2(N - 1)}) in one batch as:
exp(a(x"") nn//Ted
Q(1) ((N-1) exp(a(x") nz/Tee)
(5)
Σ
where Teon is temperature hyperparameter different from Tacl and every embedding is l2 normal- ized.
Using KL Divergence as the measure of disagreement, we can impose the consistency between the probability distributions P and Q
Lee = (KL(P()Q)
(6)
3.5 OVERALL TRAINING OBJECTIVE
Above all, the overall training objective is:
L Lee + Xz + AzCcon + adgLoe
(7)
Motivated by Huang et al. (002 we also dynamically release contrastive consistency signal. Dur- ing the first half training epochs, a is set to 0 to avoid false feature structure propagation. After half of the training epochs finish, the model becomes more stable representing feature, and gradually increase a as the epoch grows: a = Mn where T' the total number of epochs and is the current epoch number. Figure Thoos overview of our FTCC framework.
3

---

Under review as a conference paper at ICLR 2023
Sinberl:
Back Fandaider:
E-e day
prep der
Doeainey
noly the He!
Di
ale duy
Luwiee Dote
abeet
Figure Left: The overall framework of FTCC. FTCC utilizes limited labeled data to compute cross-entropy and supervised contrastive learning loss and performs augmentation on unlabeled data to compute consistency training and contrastive consistency loss. Right: The illustration of contrastive consistency. FTCC computes the distances between the original augmented views and the negatives in one batch to enforce their consistency and enhance the representation ability of the model.
4 EXPERIMENT
4.1 SET UP
We conduct our experiments on following four benehmark text classification datasets. SST2 (Socher
et al. (013) is a sentiment classification dataset of movie reviews. SUBJ (Pang & Leel 2004) is review dataset with sentences labelled as subjective or objective. PC (Ganapathibhotla & Liu| 2008) is binary sentiment classification dataset that includes Pros and Cons data. IMDB (Maas et al. 2011) is a sentiment classification dataset from IMDB movie reviews. The dataset statistics are shown in Table All datasets are in English language.
Table Statistics of four text datasets. Unlabeled and Dev dataset have even label distribution.
#Class
#Train
#Test
#Unlabeled
#Dev
SST2
2
7447
1821
3000
1000
SUBJ
2
9000
1000
3600
1000
IMDB
2
25000
25000
10000
4000
PC
2
32097
13759
12000
4000
In few-shot learning scenario, set the number of training labeled examples K 10 per class. We follow Gao et al. (2021a) to randomly split training dataset into labeled data, unlabeled data and development data in 3 different samples. During splitting, we take label distribution and the size of dataset into account and do not use all unlabeled data. Three different models are trained on these splits. Then we evaluate the average performance of these three models on given test set.
We choose German as the intermediate language to perform data augmentation with Fairseq toolkit (Ot al. 2019). We use pre-trained language model BERT as our backbone encode all examples and use the input format "[CLS] sentence [SEP]" for all models. For three datasets SST2, SUBJ, PC, the max length are set to be the first 128, 128, and 64 tokens. In IMDB dataset, we remained the last 256 tokens, as suggested by Xie et al. (2019). We use Adam (Kingma & Ba 2014) as the optimizer with linear decaying schedule. For simplicity, we set X₁, Au, Au to 1. For all datasets, the batch size of labeled data is 8 and the batch size of unlabeled data is 32, 24. 32, 32. Detailed information on experimental hyperparameter settings can be found in Table [ We run experiments one NVIDIA RTX A6000 GPU.
4

---

Under review as a conference paper at ICLR 2023
Table 2: Hyperparameter Configuration
SST2
IMDB
SUBJ
PC
max length
128
256
128
64
labeled batch size
8
8
8
8
unlabeled batch size
24
32
32
learning rate max step SCL temperature
1e-5
2e-5
le-5
le-5
2000
2500
2000
2500
0.03
0.01
0.01
0.03
CC temperature
0.1
0.1
0.1
0.1
warm-up percent
0.1
0.1
0.1
0.1
weight decay
0.01
0.01
0.01
0.01
4.2 BASELINE
We compare our FTCC to five baselines: (1) BERT-FT: Fine-tuning the pre-trained BERT-based uncased model on the labeled texts directly, (2) BERT-SCL (Gunel al. 202I): Fine-tuning BERT with cross-entropy loss and supervised contrastive loss (3) DualCL (Chen et al. 2022): Dual con- trastive learning framework learns feature representations of imput examples and corresponding clas- sifiers in the same space. It classifies documents based their similarity with classifier represen- tation (4) UDA (Xie et al., (211)): Unsupervised Data Augmentation uses limited labeled examples for supervised training and encourages model to have consistent prediction between unlabeled ex- amples and corresponding augmented examples. (5) MixText (Chen et al.) (2020a): MixText creates multiple augmented training examples by interpolating text in hidden space and then perform sistency training. To make fair comparison, we reproduce the results with the best hyperparameter configurations for all baselines.
Table 3: Performance (test accuracy ())) comparison with baselines. We use 10 labeled examples for each class. The results averaged after three random seeds show the significance Dror al. + denotes the standard error of the mean.
K= 10
SST2
IMDB
SUBJ
PC
BERT-FT
67.76+4.37
69.5814.75
86.73+1.49
75.80+2.39
BERT-SCL
70.01+4.69
66.97+1.98
88.56+2.51
80.79+2.14
DualCI.
77.32+2.82
68.89+0.25
83.03+1.36
82.75+0.94
UDA
73.43+12.05
89.06+1.50
92.70+0.70
88.5443.65
MixText
75.10+10.56
83.38+3.31
92.83+0.40
19.21
FTCC
86.43+1.21
90.1720.15
93.30:0.17
90.51+0.79
4.3
MAIN RESULT
The classification accuracy of all methods is shown in Table 3 FTCC outperforms all baselines by average. Compared to basie fine-tune baselines, FTCC outperforms them by large margin up to 20%. Compared to DualCL. FTCC does not use label-aware augmentation and instead uses back- translation to generate additional views of training examples. Both methods incorporate supervised contrastive learning while our method focuses on the usage of unlabeled data and achieves more competitive performance. Compared to UDA, FTCC discards additional training techniques (Confidence-basee masking, Sharpening Predictions (Gannvalett & Bengio (2004), Training Signal Annealing) and uses supervised contrastive learning and contrastive consistency to prevent overfit and achieve better and more stable performance. Compared to Mixtext, FTCC does not redesign pre-train language model between different layers but adopts simply but effective method to fine- tune it. which demonstrates that learning quality feature representation for data is more effective and efficient in low now-resources scenario and FTCC more robust and capable of fully leveraging labeled and unlabeled data.
5

---

Under review as a conference paper at ICLR 2023
4.4 VISUALIZATION
To explore how FTCC improves the quality of learning representations, we uses I-SNE (van der Maaten Hinton 2008 visualize learned CLS embeddings four datasets. We shows the results in Figure 2 We can find that only using 10 examples per class to fine-tune BERT with cross-entropy loss has unstructured feature representation. However, our model can clearly separate examples from different classes and pull examples from same class closer. It demonstrates that the learned representations are more discriminative and enhance our model's robustness and generaliza- tion.
(a) IMDB
PC
SST2
(d) SUBJ
Figure 2: t-SNE plots of learned sentence embeddings on four test datasets. Different colors denote different classes. In every pair, the left cross-entropy, the right is FTCC.
4.5 ABLATION STUDY
To further examine the effectiveness of each component, we investigate FTCC with the following ablations by removing component time: (1) do perform supervised contrastive learning (w/o. SCL) (2) do not perform contrastive consistency (w/o. CC) (3) do not perform supervised contrastive learning (w/o. CON). The results are shown in Table We can observe that every removal causes worse performance than original model, confirming the necessity of each loss term; Moreover, simply using SCL or CON has highly variance result while jointly incorporating them in our model design successfully alleviates this issue; CC component during the second half of training process further refines deep clustering and boosts FTCC's performance.
Table 4: Ablation study of FTCC. See abbreviation meanings in texts
Ablations
SST2
IMDB
SUBJ
PC
w/o. SCL
73.5149.26
87.0741.16
90.1342.50
86.6444.37
w/o. CC 85.62+1.53
90.14+0.14
92.73++0.49
88. 88.08+4.96
w/o. CON
78.5642.25
84.8845.73
92.40+0.17
88.84+3.12
CONCLUSION
In this paper, propose FTCC that can effectively utilize unlabeled data under few-shot scenario and simultaneously enforce more compact clustering of sentence embeddings with similar semantic meanings. Moreover, we propose contrastive consistency schema that improve the quality of feature representation and the ability of model generalization. Our model achieves competitive per- formance four text classification datasets, outperforming previous methods. Since using super- vised contrastive learning to fine-tune pre-trained language model is simple but effective training technique, the future, hope incorporate this method into weakly-supervised unsupervised NLP framework to maximally reduce human cost.
REFERENCES
Sanjeev Arora, Hrishikesh Khandeparkar, Mikhail Khodak, Orestis Plevrakis, and Nikunj Saun- shi. A theoretical analysis of contrastive unsupervised representation learning. CoRR, abs/1902.09229,, 2019. 2019.
Jiaao Chen, Zichao Yang, and Diyi Yang. MixText: finguistialll-iinomma interpolation hidden space for semi-supervised text classification. In Proceedings the 58th Annual Meeting of the

---

Under review as a conference paper at ICLR 2023
Association for Computational Linguistics, pp. 2147-2157, Online, July 2020a. Association for Computational Linguistics. doi: 10.18653/w1/2020.... mal-.m 194.
Qianben Chen, Richong Zhang, Yaowei Zheng, and Yongyi Mao. Dual contrastive learning: Text classification via label-aware data augmentation. arXiv preprint, 2022.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, 2020b.
Wei Yen Chong, Bhawani Selvaretnam, and Liy-Ki Soon. Natural language processing for sentiment analysis: An exploratory analysis on tweets. pp. 212-217, 2014. doi: 10. 109/CAAETT:2044.4
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/v 1/N19/11/1--112
Rotem Dror, Gili Baumer, Segev Shlomov, and Roi Reichart. The hitchhiker's guide to testing statistical significance in natural language processing. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Pupers), pp. 1383-1392. Association for Computational Linguisties. doi: (0.18653/v1/P1--111
Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. Understanding back-translation at scale. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 489-500. Association for Computational Linguistics, 2018. doi: 10.18653/v1/
D18- 1045.
Hongchao Fang and Pengtao Xie. CERT: contrastive self-supervised learning for language under- standing. CoRR, hlw/2005.12766. 2020.
Murthy Ganapathibhotla and Bing Liu. Mining opinions in comparative sentences. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pp. 241-248. Coling 2008 Organizing Committee, 2008
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Vol- ume 1: Long Papers), pp. 3816-3830. Association for Computational Linguistics, 2021a. doi: 10.18653/v 1L118653/112022110 1/2.21.122/2/222 aclool.acl-..n..295..
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. CoRR, abs/2104.08821, 2021b.
Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In I.. Saul, Y. Weiss, and L. Bottou (eds.), Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004.
Alex Graves and Jürgen Schmidhuber. Framewise phoneme classification with bidirectional Istm and other neural network architectures. Neural Networks, 18(5):602-610, 2005. ISSN 0893- 6080. doi: bttps::/doi.org//10
IJCNN 2005.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Veselin Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. In International Conference on Learning Represen- tations, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. unsupervised visual representation learning, 2019.
Momentum contrast for
Jiaxin Huang, Yu Meng, and Jiawei Han. Few-shot fine- -eaiggrannd entity typing with automatic label interpretation and instance generation, 2022.

---

Under review as a conference paper at ICLR 2023
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Pro- cessing Systems, volume 33, pp. 18661-18673. Curran Associates, Inc., 2020.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach, arXiv preprint orxi 1692, 2019.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng. and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150. Association for Computational Linguisties, 2011.
Takeru Miyato, Andrew M. Dai, and Ian Goodfellow. Adversarial training methods for semi-
supervised text classification, 2016.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: A regularization method for supervised and semi-supervised learning, 2017.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng. David Grangier, and Michael Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics (Demonstrations), pp. 48-53. Association for Computational Linguistics, 2019. doi: 10. 00665//////111...0
Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summa- rization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04), pp. 271-278, 2004. doi: 10.3115/12189553 1218999....
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp. 2383-2392 Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10. 20118653vv11D/ 1264.
G. M. Shahariar, Swapnil Biswas, Faiza Omar, Faisal Muhammad Shah, and Samiha Binte Hassan. Spam review detection using deep learning. In 2019 IEEE 10th Annual Information Technology, Electronics and Mobile Communication Conference (IEMCON), pp. 0027-0033, 2019. doi: 10. 1109/EMCON.201 2019 8936148.
Richard Socher, John Bauer, Christopher D. Manning, and Andrew Ng. Parsing with composi- tional vector grammars. In Proceedings of the 51st Annual Meeting of the Association for Com- putational Linguistics (Volume Long Papers), pp. 455-465. Association for Computational Linguisties, 2013.
Amane Sugiyama and Naoki Yoshinaga. Data augmentation using back-translation for context- aware neural machine translation. Proceedings of the Fourth Workshop on Discourse in Ma- chine Translation (DiscoMT' 2019), pp. 35.44. Association for Computational Linguistics, 2019. doi: DL.1863//1/D199-66
Laurens van der Maaten and Geoffrey Hinton. Viualizing data using t-sne. Journal of Machine Learning Research, 9:2579-2605, 11 2008.
Dong Wang, Ning Ding, Piji Li, and Hai-Tao Zheng. CLINE: contrastive learning with semantic negative examples for natural language understanding. CoRR, abs/2107 17.004440 2021.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. In Proceedings of the 37th International Conference Machine Learning, ICML'20. JMLR.org, 2020.
Chen Wei, Huiyu Wang, Wei Shen, and Alan Yuille. {CO)22 Consistent contrast for unsupervised visual representation learning. In International Conference on Learning Representations, 2021.
8

---

Under review as a conference paper at ICLR 2023
Qizhe Xie. Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. Unsupervised data augmentation for consistency training. arXiv preprint arXiv:1904. 12848, 2019.
Yuanmeng Yan, Rumei Li, Sirui Wang, Fuzheng Zhang, Wei Wu, and Weiran Xu. ConSERT: A con- trastive framework for self-supervised sentence representation transfer. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 5065 -0065,505,, Online, August 2021. I. Association for Computational Linguistics. doi: 10. 01886533v122222 Lacl- aon-1ong.393.


Reviews:
[{"review_id": "rCFR8NaBGlL", "reviewer": "Reviewer_CtgC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "\nThe paper considers few-shot text classification for pre-trained LMs. A supervised contrastive learning approach is taken, with consistency-regularization, and evaluation is performed across SST2, IMDB, SUBJ and PC datasets, against a collection of recent models, including BERT-SCL and DualCL. \n", "summary_of_the_review": "\nContrastive, few-shot learning in text is a popular area of research. This paper’s novelty is around consistency-regularization applied to this scenario, but this may not be sufficient, and overly simplistic empirical approaches should be updated with closer baselines, additional datasets, and more rigorous ablations. ", "strength_and_weaknesses": "Strengths\n\n- contrastive learning for few-shot learning in text is of course an interesting solution.\n- the datasets chosen are standard for text-classification tasks, and the baselines are fine, generic approaches (although other baselines closer to the proposed method could have been selected, as per below)\n- the empirical results (Table 3) are, of course, far better for the proposed method than the alternatives, across all selected datasets.\n\nWeaknesses\n\n- Few-shot learning in NLP is a very well explored problem and some key papers in this area, especially more recent ones than those cited (e.g., arxiv:2005.14165) are not considered.\n- The benchmarking datasets of RAFT and ODIC should have been at least mentioned if not used.\n- The ablation studies need to be better explained, within an overall better empirical framework. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "oncf6fWq_9c", "reviewer": "Reviewer_6ZcP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first claims that in few-shot settings, fine-tuning pre-trained language models doesn’t achieve satisfactory performances. Therefore, this paper proposes a method to employ contrastive learning for few-shot fine-tuning of pre-trained models, due to the success of contrastive learning in other few-shot scenarios. The experiments are only conducted on pre-trained BERT-base model on 4 downstream NLP tasks.", "summary_of_the_review": "please see under Strength and Weaknesses.", "strength_and_weaknesses": "My main concern with this submission is that it overlooks too much relevant work in this field. First of all, it is not true that all pre-trained language models are horrible few-shot learners. Actually some large pre-trained models, especially the auto-regressive ones, achieve quite surprising few-shot performance[1,2,3,4]. Second, a lot of existing work also shows that proper prompts[4,5] and proper fine-tuning methods[6,7] can also largely improve pre-trained models’ few-shot performance in general. Therefore, I think without discussing and comparing with such methods, the effectiveness or novelty of the proposed method are unclear.\n\nMy second concern is on the experiment side. As mentioned earlier, the method is only applied on BERT-base. More pre-trained models should be considered, especially those having good few-shot performances. Moreover, the number of shots, 10, is a bit random to me as well. Diverse numbers of shots need to be tested.  3 random seeds also sound too small for me because of the huge variance of few-shot experiments. Please also follow existing work to increase the number of random seeds[7]. \n\n[1] Brown, Tom, et al. \"Language models are few-shot learners.\" Advances in neural information processing systems 33 (2020): 1877-1901.\n\n[2] Raffel, Colin, et al. \"Exploring the limits of transfer learning with a unified text-to-text transformer.\" J. Mach. Learn. Res. 21.140 (2020): 1-67.\n\n[3] Radford, Alec, et al. \"Language models are unsupervised multitask learners.\" OpenAI blog 1.8 (2019): 9.\n\n[4]Gao, Tianyu, Adam Fisch, and Danqi Chen. \"Making pre-trained language models better few-shot learners.\" arXiv preprint arXiv:2012.15723 (2020).\n\n[5]Perez, Ethan, Douwe Kiela, and Kyunghyun Cho. \"True few-shot learning with language models.\" Advances in Neural Information Processing Systems 34 (2021): 11054-11070.\n\n[6]  Zhang, Ningyu, et al. \"Differentiable prompt makes pre-trained language models better few-shot learners.\" arXiv preprint arXiv:2108.13161 (2021).\n\n[7] Mahabadi, Rabeeh Karimi, et al. \"Perfect: Prompt-free and efficient few-shot learning with language models.\" arXiv preprint arXiv:2204.01172 (2022).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "naIE-LcyxR", "reviewer": "Reviewer_3uH4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novel Contrastive Consistency (CC) learning objective for few-shot text classification in semi-supervised settings (i.e., when additional unlabeled data are available). This new CC loss term is added when fine-tuning a BERT model, together with:\n- a Supervised Constrastive Learning (SCL) loss (from Gunel et al, 2021)\n- Consistency Training (CON) (adapted from Xie et al., 2019), \non top of the regular cross-entropy. \n\nThe authors show their method outperforms 4 recent baselines on 4 English datasets.\n\n\n", "summary_of_the_review": "This paper is an easy and interesting read that could be useful to some ML practitioners. Its novelty seems however fairly limited and as such the paper feels short of the bar for ICLR. Since this paper contribution is mostly experimental, its experiment section could perhaps be more comprehensive and provide additional insights.", "strength_and_weaknesses": "Overall, the paper provides interesting insights on simple ways to improve few-shot classification for text applications.\n\nThe novelty of this paper mostly lies in combination of various losses from the literature. Of the 3 additional loss terms used by the authors, two are applied as-is (modulo minor details) from existing papers and one is adapted from computer vision. Empirically, this combination is shown to allow improving classification accuracy and reducing variance.\n\nWhile the paper is generally clear, the paper feels like a short paper. For an ICLR paper, I would have liked a deeper motivation and review section, as well as additional experiments and insights. For example:\n- how do the results change when the number of labeled examples vary (e.g., 3, 10, 50, 100)?\n- any theoretical/empirical reason to set all lambdas to 1?\n- can the ablation study be better explained? I interpreted CC as the main contribution of the paper, but its impact seem minor relative to the both SCL and CON.\n\nFinally, the exact differences, or lack thereof, between the losses in FTCC and other papers could at times be further clarified inside section 3.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "anpFLuWgodF", "reviewer": "Reviewer_auWg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a text classification method that adds three regularization constraints. The first is a contrastive term between labeled samples depending on whether pairs of inputs have the same labels. The second is a consistency constraint using K-L divergency on back-translated transformation of the data samples. The third is a K-L divergence on the contrastive distribution between data samples and their back-translated transformations. The latter two can be performed on unlabeled data.\n\nExperiments are performed on four binary classification tasks with number of training samples up to 32097 and number of unlabeled samples up to 12000. Superior results are observed compared to BERT and other data augmentation baselines. t-SNE analysis on the results are performed to show that the model obtains clear classification boundary. Ablation study is performed for each of the regularization contraints to show their respective necessity in improving the results.", "summary_of_the_review": "Idea has good novelty. Execution needs improvement. Datasets for experiments are limited.", "strength_and_weaknesses": "Strengths of the paper:\n1) The idea of using back-translation for regularizing text classification models is a solid one.\n2) The three proposes regularization constraints are reasonable.\n3) Superior results compared to BERT and other baselines. In the literature, it is very hard to beat the BERT model for text classification tasks.\n4) Good ablation study to demonstrate how each of the constraints are useful.\n\nWeaknesses of the paper:\n1) Only binary classification tasks where chosen. There are a large number of richer text classification datasets available in the literature.\n2) The datasets used are very small. We do not know whether the model can still perform well when large number of training samples are available.\n3) Traditional linear classifiers using bag-of-words, n-gram TFIDF etc is missing for the baselines. Along with BERT, these baselines are strong therefore should be considered in the study.\n4) The ablation study should include a baseline classification result without any of the constraint, so that the readers can be clearer about the relative performance improvement.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject