SlowGuess/NAACL_2022_OCR huggingface dataset:
Meta re-weighting leverages gradient as a proxy to estimate the weight of noisy pseudo labels. However, the gradients of adapter parameters  $\psi$  are not stable in the early stages of training due to random initialization and noises in pseudo labels. This instability issue is further exacerbated with adapter tuning that usually requires a larger learning rate (Pfeiffer et al., 2020). Therefore, to stabilize adapter tuning, we propose a warmup training stage via knowledge distillation (Hinton et al., 2015) to first tune adapter parameters via knowledge distillation loss for  $T_{warm}$  steps and then we continue self-training with re-weighted updates via Eq. 7. Since the re-weighting procedure has access to our training labels, we do not use labeled data in knowledge distillation while using only the unsupervised consistency loss between teacher model  $(\Theta_{\mathrm{PLM}},\hat{\psi}_{\mathrm{tea}})$  and student model  $(\Theta_{\mathrm{PLM}},\hat{\psi}_{\mathrm{stu}})$  on unlabeled data as.
$$
\underset {\hat {\psi} _ {s t u}} {\arg \min } \mathrm {K L} \left(f \left(\widetilde {x} ^ {u}; \Theta_ {\mathrm {P L M}}, \hat {\psi} _ {\text {t e a}}\right) \| f \left(\widetilde {x} ^ {u}; \Theta_ {\mathrm {P L M}}, \hat {\psi} _ {\text {s t u}}\right)\right). \tag {8}
$$
We further validate the effectiveness of knowledge distillation for warmup with ablation analysis.
4.3.2 Student Adapter Re-initialization
A typical challenge in few-shot settings is the lack of a separate validation set. In the spirit of true few-shot learning, we use only the available few-shot labeled examples  $\widetilde{\mathcal{D}}_K^{Train}$  as the validation set for meta-learning of the student model. This poses an interesting challenge of preventing label leakage. To address this issue, we re-initialize the student adapter parameters every time at the start of each self-training iteration to mitigate interference with labeled data. Note that the student and teacher model share the encoder parameters  $\Theta_{\mathrm{PLM}}$  that are always kept frozen and not updated during training.
Dataset. We perform large-scale experiments with six natural language understanding tasks as summarized in Table 6. We use four tasks
from GLUE (Wang et al., 2019), including MNLI (Williams et al., 2018b) for natural language inference, RTE (Dagan et al., 2005; Bar Haim et al., 2006; Giampiccolo et al., 2007; Bentivogli et al., 2009) for textual entailment,  $\mathbf{QQP^4}$  for semantic equivalence and SST-2 (Socher et al.) for sentiment classification. The results are reported on their development set following (Zhang et al., 2021). MPQA (Wiebe et al., 2005) and Subj (Pang and Lee, 2004) are used for polarity and subjectivity detection, where we follow (Gao et al., 2021) to keep 2,000 examples for testing and use remaining examples for semi-supervised learning.
For each dataset, we randomly sample  $|\mathcal{K}| \in \{10,20,30\}$  manually labeled samples from the training data, and add the remaining to the unlabeled set while ignoring their labels - following standard setups for semi-supervised learning. We repeatedly sample  $K$  labeled instances five times, run each model with 5 different seeds and report average performance with standard deviation across the runs. For the average accuracy over 6 tasks, we did not include standard deviation across tasks. Furthermore, for every split and shot, we sample the labeled data such that  $\mathcal{D}_{10}^{Train} \subset \mathcal{D}_{20}^{Train} \subset \mathcal{D}_{30}^{Train}$  to evaluate the impact of incremental sample injection.
Following true few-shot learning setting (Perez et al., 2021), we do not use additional development set beyond  $|\mathcal{K}|$  labeled samples for any hyperparameter tuning or early stopping. The performance of each model is reported after fixed training epochs (see Appendix for details).
Baselines. In addition to classic-tuning (Classic FN), we adopt prompt-based fine-tuning (Prompt FN) from (Gao et al., 2021) as labeled-only baselines. We also adopt several state-of-the-art semi-supervised baselines including UST (Mukherjee and Awadallah, 2020), MetaST (Wang et al., 2021c) and iPET (Schick and Schütze, 2021a). UST and MetaST are two self-training methods which are based on classic fine-tuning strategies. iPET is a semi-supervised method leveraging prompt-based fine-tuning and prompt ensembles to obtain state-of-the-art performance. While iPET ensembles multiple fully-tuned models, we develop a lite self-training framework to achieve both data and parameter efficiency. As the strongest semi-supervised baseline, we implement a new method PromptST based on self-training using prompts and adapters
4https://www.quora.com/q/quoradata/
Labels	Models	Avg	#Tunable
Params	MNLI (m/mm)
(acc)	RTE
(acc)	QQP
(acc)	SST-2
(acc)	Subj
(acc)	MPQA
(acc)
|K|=30	Classic FN	60.9	355M	38.0 (1.7) / 39.0 (3.1)	51.4 (3.7)	64.3 (8.1)	65.0 (11.5)	90.2 (2.2)	56.1 (5.3)
Prompt FN	77.6	355M	62.8 (2.6) / 64.1 (3.3)	66.1 (2.2)	71.1 (1.5)	91.5 (1.0)	91.0 (0.5)	82.7 (3.8)
|K|=30 +Unlabeled Data	UST	65.8	355M	40.5 (3.3) / 41.5 (2.9)	53.4 (1.7)	61.8 (4.3)	76.2 (11.4)	91.5 (2.1)	70.9 (6.2)
MetaST	62.6	355M	39.4 (3.9) / 40.5 (4.4)	52.9 (2.0)	65.7 (6.2)	65.3 (15.2)	91.4 (2.3)	60.5 (3.6)
iPET	75.5	355M	61.0 (5.8) / 61.8 (4.7)	54.7 (2.8)	67.3 (4.1)	93.8 (0.6)	92.6 (1.5)	83.1 (4.8)
PromptST	77.2	14M	61.8 (1.9) / 63.1 (2.9)	66.2 (5.1)	71.4 (2.1)	91.1 (1.4)	90.3 (1.5)	81.8 (2.5)
LiST	82.0	14M	73.5 (2.8) / 75.0 (3.7)	71.0 (2.4)	75.2 (0.9)	92.8 (0.9)	93.5 (2.2)	85.2 (2.1)
Supervision with # Full Train	Classic FN	90.9	355M	89.6 / 89.5	83.0	91.8	95.2	97.2	88.8
Prompt FN	92.0	355M	89.3 / 88.8	88.4	92.1	95.9	97.1	89.3
Table 1: Performance comparison of different tuning strategies on different NLU tasks with RoBERTa-large as the encoder with standard deviation in parentheses. UST, MetaST, PromptST and iPET are semi-supervised methods using unlabeled data, whereas Classic and Prompt FN only use labeled data.
(as a subset of the methods used in LiST), but without any re-weighting, or KD warmup that are additionally used in LiST. The methods PromptFN, PromptST and LiST adopt same prompts and label words as in (Gao et al., 2021). We implement our framework in Pytorch and use Tesla V100 gpus for experiments. Prompts used in experiments and hyper-parameter configurations are presented in Appendix.
5.2 Key Results
Table 1 shows the performance comparison among different models with  $|\mathcal{K}| = 30$  labeled examples with fixing RoBERTa-large as the encoder. Fully-supervised RoBERTa-large trained on thousands of labeled examples provides the ceiling performance for the few-shot setting. We observe LiST to significantly outperform other state-of-the-art baselines along with  $96\%$  reduction in tunable parameters, achieving both labeled data- and parameter-efficiency. More specifically, LiST improves over Classic FN, Prompt FN, iPET and PromptST by  $34.6\%$ ,  $5.7\%$ ,  $8.6\%$  and  $6.2\%$  respectively in terms of average performance on six tasks. This demonstrates the impact of self-training with unlabeled data and prompt-based FN. Additionally, iPET and LiST both leverage prompt-based FN to significantly improve over UST and MetaST that use classic fine-tuning strategies, confirming the effectiveness of prompt-based FN in the low data regime. iPET ensembles multiple prompts with diverse qualities and under-performs Prompt FN on average in our few-shot setting without using any development set.
Figure 5 compares the performance of tuning methods with varying number of training labels
(a) MNLI
(b) RTE
Figure 5: Performance comparison of Classic-tuning (denoted as "C") and prompt-based fine-tuning (denoted as "P") with List on MNLI and RTE using language model encoders of different sizes.
and encoders of different sizes. We observe that large models are more data-efficient compared to smaller models. However, large fully-tunable models are expensive to use in practice. We observe that LiST with small number of tunable parameters consistently outperforms fully-tunable classic and prompt-based FN strategies in all labeled data settings, demonstrating both data and parameter ef
Labels	Fine-tuning Method	Avg	MNLI (m/mm) (acc)	RTE (acc)	QQP (acc)	SST-2 (acc)	Subj (acc)	MPQA (acc)
|K|=10	GPT-3 In-context	61.5	36.4 (0.8) / 36.7 (1.3)	53.2 (1.8)	61.8 (3.0)	86.6 (7.4)	61.0 (11.2)	66.7 (9.5)
Prompt-based FN	69.3	54.8 (3.7) / 55.6 (4.6)	60.0 (4.4)	58.7 (4.6)	89.5 (1.7)	84.5 (8.6)	67.8 (6.9)
LiST	72.8	62.6 (6.6) / 63.3 (7.7)	61.2 (4.9)	60.4 (7.0)	91.1 (1.2)	91.0 (1.6)	70.3 (10.6)
|K|=20	GPT-3 In-context	57.4	38.0 (2.0) / 38.4 (2.8)	54.5 (1.5)	64.2 (1.6)	79.1 (2.3)	51.2 (1.7)	72.4 (8.5)
Prompt-based FN	75.4	60.3 (2.0) / 61.6 (2.7)	64.3 (2.4)	67.8 (4.2)	90.6 (1.8)	88.3 (2.2)	80.6 (7.5)
LiST	79.5	68.9 (3.1) / 70.4 (3.3)	69.0 (3.5)	72.3 (3.7)	92.3 (1.2)	91.5 (1.3)	82.2 (5.1)
|K|=30	GPT-3 In-context	61.5	37.9 (2.2) / 38.5 (2.9)	53.4 (2.2)	65.0 (1.7)	79.7 (7.1)	57.7 (6.4)	74.8 (6.9)
Prompt-based FN	77.6	62.8 (2.6) / 64.1 (3.3)	66.1 (2.2)	71.1 (1.5)	91.5 (1.0)	91.0 (0.5)	82.7 (3.8)
LiST	82.0	73.5 (2.8) / 75.0 (3.7)	71.0 (2.4)	75.2 (0.9)	92.8 (0.9)	93.5 (2.2)	85.2 (2.1)
Table 2: Average performance and standard deviation of GPT-3 (175B params) in-context learning, Prompt-based FN and LiST methods using Roberta-Large (355M params) encoder with varying number of training labels  $|\mathcal{K}|$ . LiST updates 14M params in contrast to Prompt-based FN with full model tuning.
Tuning	#Params	Avg	Diff
Full	355M	77.6	—
Embedding	53M	67.0	-10.7
Attention	101M	77.0	-0.6
FF-output	102M	77.6	+0.0
FF-intermediate	102M	75.9	-1.7
ficiency. Additional results with different backbone encoders and varying number of shots and finetuning strategies are presented in the Appendix in Tables 13, 14, 15 and 19 that demonstrate similar trends as we observe in Table 1 and Figure 5.
Comparison with GPT-3 in-context Learning. We perform a comparison between GPT-3 in-context learning, RoBERTa-large Prompt-based fine-tuning and LiST methods with varying number of training labels in Table 2. For a fair comparison, the prompt and label words are same for the three approaches. We observe that LiST outperforms GPT-3 In-context learning and Prompt-based FN consistently with different number of labels.
5.3 Adapter Analysis
In this section, we explore adapter design choices for prompt-based FN with RoBERTa-large as encoder using only few-shot labeled data.
Where to insert an adapter in Transformers? In order to answer this question, we conduct an experiment to study the role of various Transformer modules in few-shot prompt-based FN. To this end, we tune a given module along with the language model head while keeping all other parameters frozen. Ta
Table 3: Average accuracy on tuning different modules of RoBERTa-large with  $|\mathcal{K}| = 30$  labels on six tasks. Diff shows performance change relative to Full tuning.
Tuning	#Params	Avg
Head-only	1M	66.9
Bias-only (Cai et al., 2020b)	1M	68.3
Prompt-tuning (Lester et al., 2021b)	1M	56.4
LiST Adapter (2)	1M	72.7
Houlsby Adapter (Houlsby et al., 2019)	14M	57.9
LiST Adapter (128)	14M	77.7
Full tuning	355M	77.6
Table 4: Average accuracy of several lightweight parameter-efficient tuning strategies with  $|\mathcal{K}| = 30$  labels without unlabeled data on six tasks along with the number (#) of tunable parameters. Each task is run with 5 different seeds. List Adapter performance with different bottleneck dimension  $d$  of its adapters is shown in parentheses.
ble 3 shows the performance comparison of tuning specific modules on six tasks with varying number of labeled examples. The main modules of RoBERTa include Embedding, Attention, Feedforward Output and Feedforward Intermediate layers. We observe that tuning only the Feedforward Output or the Attention module delivers the best performance across most tasks with few-shot labels. Correspondingly, this motivated us to insert our adapter parameters into these two modules. More detailed results are presented in Appendix Table 11. Comparison with other lightweight parameter efficient model tuning strategies. To validate the effectiveness of LiST adapters, we compare it against several baselines in Table 4. For a fair comparison, we present two variations of our LiST adapters with bottleneck dimensions  $d = \{2,128\}$  corresponding to  $1M$  and  $14M$  parameters to match other adapter capacities; all the approaches in Table 4 are trained with 30 labels only without
unlabeled data for a fair comparison. (1) Bias-only is a simple but effective lightweight method, which tunes bias terms of PLMs while keeping other parameters frozen. (2) Tuning head layers is widely used as a strong baseline for lightweight studies (Houlsby et al., 2019), where we tune last two layers including language model head while freezing other parameters. (3) prompt-tuning is a lightweight method which only updates task prompt embedding while keeping entire model frozen. (4) Houlsby Adapter tunes inserted adapter parameters keeping the encoder frozen by adopting classic tuning strategy. Besides these lightweight methods, we also present a performance comparison with full model tuning as a strong baseline. More detailed results are presented in Appendix in Tables 12 and 20 that demonstrate similar trends.
Table 4 shows that LiST is able to match the performance of full model prompt-based FN with bottleneck dimension  $d = 128$  and outperforms all other baselines with similar capacities. While lightweight model tuning choices like tuning the bias or inserting adapters into classic tuning models are shown to be effective in fully-supervised settings (Cai et al., 2020b; Houlsby et al., 2019), we observe them to under-perform for few-shot learning. We observe that simpler tuning choices like Head-only and Bias-only results in up to  $10\%$  performance degradation. Houlsby adapter and Prompt-only results in up to  $20\%$  performance degradation. In constraint, LiST adapter is able to match the performance of full tuning in few-shot setting, demonstrating the importance of adapter placement choices and parameter initialization.
Table 5 demonstrates the impact of different components and design choices of LiST.
- Adapter training stability. Training with very few labels and noisy pseudo labeled data results in instability for adapter tuning. To demonstrate training stability, we include the average accuracy and standard deviation across several runs and splits as metrics. We observe that hard pseudo-labels hurt the model performance compared to soft pseudolabels and exacerbate the instability issue. This is in contrast to observations from classic finetuning (Wang et al., 2021c). A potential reason could be that the well pre-trained language model head for prompt-based FN is able to capture better associations among different prompt labels.
- Knowledge Distillation Warmup. In this ab
Method	Avg Acc	Avg Std	Datasets
MNLI (m/mm)	RTE
LiST (14M)	72.6	2.8	73.5 (2.8) / 75.0 (3.7)	71.0 (2.4)
w/o re-init	68.3	4.2	66.7 (2.8) / 68.3 (4.3)	69.0 (4.9)
w/o KD Warmup	68.8	8.8	67.9 (12.9) / 69.0 (13.1)	69.2 (4.5)
w/o Re-weighting	71.6	4.0	72.9 (3.4) / 74.2 (4.5)	69.7 (4.1)
w/ Hard Pseudo-Labels	70.9	4.4	71.7 (3.8) / 73.0 (5.4)	69.5 (4.2)
LiST w/o Adapter (355M)	72.6	2.5	73.6 (2.7) / 74.8 (2.7)	71.2 (2.3)
Table 5: Ablation analysis of LiST with 30 labels on MNLI and RTE with tunable parameters in parentheses.
lation study, we remove the warmup phase with knowledge distillation from LiST (denoted as "LiST w/o KD Warmup"). Removing this component results in  $4\%$  performance drop in terms of average accuracy and  $300\%$  larger standard deviation - demonstrating the importance of KD Warmup in stabilizing LiST training.
- LiST versus LiST w/o Adapter. In LiST, we only fine-tune the adapter and language model head while keeping other encoder parameters frozen to achieve parameter efficiency. Table 5 shows that LiST using only  $4\%$  tunable parameters is able to match the performance of fully tunable LiST (that is without using any adapters and tuning all encoder parameters) on MNLI and RTE - demonstrating the effectiveness of our lightweight design. More ablation results with varying shots are presented in Appendix in Tables 16, 17 and 18 that demonstrate similar trends as in Table 5.
We develop a new method LiST for lightweight tuning of large language models in few-shot settings. LiST uses prompted self-training to learn from large amounts of unlabeled data from target domains. In order to reduce the storage and training cost, LiST tunes only a small number of adapter parameters with few-shot labels while keeping the large encoder frozen. With only 30 labels for every task, LiST improves by upto  $35\%$  over classic fine-tuning and  $6\%$  over prompt-tuning while reducing  $96\%$  of the tunable parameters. With significant reduction in the cost of (data) annotation and overall model footprint, LiST provides an efficient framework towards life-long learning of AI agents (Biesialska et al., 2020). While adapters reduce storage cost, LiST does not reduce inference latency given the PLM backbone. A future work is to consider combining model compression techniques (Han et al., 2015; Cai et al., 2020a) with adapters to reduce FLOPS and latency.
In this work, we introduce a lightweight framework for self-training of language models with only a few labeled examples. We expect that progress and findings presented in this paper could further benefit NLP applications with limited labeled data. In the real-world setting, it is usually not only expensive to obtain large-scale labeled data for each task but also brings privacy and compliance concerns when large-scale data labeling is needed. The privacy concerns could be further exacerbated when dealing with sensitive user data for peronslization tasks. Our framework which only needs few-shot labeled data could help in this to obtain state-of-the-art-performance while alleviating privacy concerns. The proposed framework is tested across different tasks and could be used for applications in various areas including finance, legal, healthcare, retail and other domains where adoption of deep neural network may have been hindered due to lack of large-scale manual annotations on sensitive user data.
While our framework advances the progress of NLP, it also suffers from associated societal implications of automation ranging from job losses for workers who provide annotations as a service as well as for other industries relying on human labor. Moreover, it may bring additional concerns when NLP models are used by malicious agents for propagating bias, misinformation and indulging in other nefarious activities. However, many of these concerns can also be alleviated with our framework to develop better detection models and mitigation strategies with only a few representative examples of such intents.
The proposed method is somewhat compute- intensive as it involves large-scale language model. This might impose negative impact on carbon footprint from training the described models. In order to reduce the storage and training cost, the proposed design tunes only a small number of adapter parameters with few-shot labels while keeping the large encoder frozen.
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Pro
cessing (Volume 1: Long Papers), pages 7319-7328, Online. Association for Computational Linguistics.
Roy Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second PASCAL recognising textual entailment challenge.
Tilman Beck, Bela Bohlender, Christina Viehmann, Vincent Hane, Yanik Adamson, Jaber Khuri, Jonas Brossmann, Jonas Pfeiffer, and Iryna Gurevych. 2021. Adapterhub playground: Simple and flexible few-shot learning with adapters. arXiv preprint arXiv:2108.08103.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth PASCAL recognizing textual entailment challenge. In TAC.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R Costa-jussa. 2020. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6523-6541.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. 2020a. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. 2020b. Tinytl: Reduce memory, not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 33.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment.