Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding
Authors: Jianing Wang
Qiushi Sun
Nuo Chen
Chengyu Wang
Jun Huang
Ming Gao
Xiang Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Self-Training
Uncertainty Estimation
Pre-trained Language Models
Parameter-Efficient Learning
TL;DR: We presents UPET, a novel Uncertainty-aware Parameter-Efficient self-Training framework to effectively and efficiently address the problem of labeled data scarcity.

Abstract: The recent success of large pre-trained language models (PLMs) heavily hinges on massive labeled data, which typically produces inferior performance in low-resource scenarios. To remedy this dilemma, we study self-training as one of the predominant semi-supervised learning (SSL) approaches, which utilizes large-scale unlabeled data to generate synthetic examples. However, too many noisy labels will hurt the model performance, and the self-training procedure requires multiple training iterations making it more expensive if all the model parameters of the PLM are updated. This paper presents UPET, a novel Uncertainty-aware Parameter-Efficient self-Training framework to effectively and efficiently address the labeled data scarcity issue. Specifically, we incorporate Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estimation for the teacher model and then judiciously select reliable pseudo-labeled examples based on confidence and certainty. During the student training, we introduce multiple parameter-efficient learning (PEL) paradigms that allow optimizes only a small percentage of parameters. We also propose a novel Easy-Hard Contrastive Tuning to enhance the robustness and generalization. Extensive experiments over multiple downstream tasks demonstrate that UPET achieves a substantial improvement in terms of performance and efficiency. Our codes and data are released at https: //github.com/wjn1996/UPET.

Uncertainty-aware Parameter-Efficiet Self-training for Semi-supervised Language Understanding
Jianing Wang"
Qiushi Sun 00
Nuo Chen
Chengyu Wang Jun Huang"
Ming Gao
Xiang Li
School of Data Science and Engineering, East China Normal University
National University of Singapore "Alibaba Group
KLATASDS-MOE, School of Statistics, East China Normal University
Lyg com, (qiushisun, nuochen dustuu.ecce www.eccuuuuuuu...oooooooooe (chengyu. way, junhuang hj aatalibbba-inne com, mgao@ ecnu. edu. cn
Abstract
by pre-training over large-scale unsupervised cor- pora and fine-tuning the target tasks. However, the conventional fine-tuning approaches heavily de- pend on the time-consuming and labor-intensive process of data annotation, which could be even more bothersome in some real-world scenarios and typically produces inferior performance in fho-shot settings (Liu et al., 2021b; Kojima et al., 2022). Recently, self-training (Chawla and Karakoulas, 2005; Amini et al., 2022) has been presented to ad- dress the labeled data scarcity issue by leveraging the large-scale unlabeled data in addition labeled data, which is one of the mature paradigms in semi- supervised learning (Qi and Luo, 2022: Yang et al.. 2021a; Chawla and Karakoulas, 2005; van Engelen and Hoos, 2020; Yang et al., 2021b). A teacher model is fine-tuned on the few-shot labeled data, then the pseudo label of each unlabeled example can be generated. After that, student model can learn the knowledge derived from the large-scale pseudo-labeled data, leading to better performance near to full-supervised learning. Previous works typically use self-training in conjunction with large PLMs to endow the model with the ability of few- shot learning. Despite the big success, we observe that there are still two challenges. 1) The pseudo- labeled data consists of too many noises, inevitably degrading the model performance due confirma- tion bias (Wang et al., 2021). The procedure of self-training is too expensive when updating all parameters of the large PLM 1 (Wang et al., 2022).
The recent success of large pre-trained lan- guage models (PLMs) heavily hinges mas- sive labeled data, which typically produces infe- rior performance in low-resource scenarios. To remedy this dilemma, we study self-training as one of the predominant semi- superviue learn- ing (SSL) approaches, which utilizes large- scale unlabeled data to generate synthetic exam- ples. However, too many noisy labels will hurt the model performance. and the self- self-training procedure requires multiple training iterations making it more expensive if all the model pa- rameters of the PLM are updated. This pa- per presents UPET, a novel Uncertainty-aware Parameter- Efficient self- Training framework to effectively and efficiently address the labeled data scarcity issue. Specifically, we incorporate Monte Carlo (MC) dropout in Bayesian neural network (BNN) to perform uncertainty estima- tion for the teacher model and then judiciously select reliable pseudo-labeled examples based on confidence and certainty. During the stu- dent training. introduce multiple parameter- efficient learning (PEL) paradigms that allow the optimization of only a small percentage of parameters. We also propose novel Easy- Hard Contrastive Tuning to enhance the robust- ness and generalization. Extensive experiments over multiple downstream tasks demonstrate that UPET achieves substantial improve- ment in terms of performance and efficiency. Our codes and data are released at https: //gi thub. com/ w ijnn 9 996/UPTT.
Fortunately, parameter-efficient learning (PEL) opens up the possibility of attaining near statc-of- the-art performance, whilst adding only few pa- rameters per task (Mao et al., 2022; Ding et al.. 2023; Zhang et al., 2023). Notable PEL-based methods include Ptuning (Liu et al., 2021b). Prefix-
1 Introduction
Pre-trained language models (PLMs) have be- come the imperative infrastructure in a series of downstream natural language understanding (NLU) tasks (Devlin et al., 2019; Liu et al., 2019; Yang et al., 2019), aiming at capturing prior knowledge
'Generally, the number of training pscudn-labeled data for the student model is larger than labeled data.
Corresponding Author.

---

tuning (Li and Liang. 2021), Adapter (Houlsby al., 2019), BitFit (Zaken al., 2022), LoRA (Hu et al., 2022), etc. Yet, is unclear how these PEL- based methods can be applied to self-training. In this paper. we develop novel Uncertainty- aware Parameter-Efffiiint self-Training frame- work (UPET) for improving self-training through two perspectives, i.e., effectiveness and efficiency. To reach these goals, we respectively present two novel techniques, including Reliable Example Sam- pling (RES) and Efficient Robust Tuning (ERT). The goal of RES is to explicitly mitigate the effect of label noises. Concretely, we obtain the predic- tion probability distribution over all unlabeled data derived from the teacher model. Then, we utilize Monte Carlo (MC) dropout technique in Bayesian neural network (BNN) (Gal and Ghahramani, 2016; Wang and Yeung, 2016) to estimate the uncertainty of each unlabeled example. To this end, the exam- ple with higher confidence and certainty will be judiciously selected as the reliable pseudo-labeled data. In ERT, we aim to leverage PEL paradigms to train robust student model over reliable pscudo- labeled data. We design multiple PEL-based model architectures for the student model that only need to update small scope of tunable parameters in PLM during itcrative trainiia Additionally, we in- troduce Easy-Hard Contrastive Tuning to improve the robustness of the parameter-efffcient model, which can be viewed regularization in the se- mantic space that keeps the noisy labels away from the reliable examples.
addition to labeled data, which has been widely used in the NLP community (Yang et al., 2017; Gururangan et al., 2019; Xie et al., 2020; Chen et al., 2020). For instance, Yang et al. (2017); Gu- rurangan et al. (2019) utilize variational autoen- coders (VAEs) for sequence classification and la- beling. Chen et al. (2020) proposes MixText to mix labeled, unlabeled, and augmented data, and performs similar consistency training UDA (Xie et al., 2020). Self-training is one of the mature SSL approaches that use teacher-student architecture to augment data (Hu and Khan, 2021; Mukherjee and Awadallah, 2020; Amini et al., 2022; Wang et al., 2021; Tsai et al., 2022). For example, Hu and Khan (2021) presents uncertainty estimation for de- noising self-training. Tsai et al. (2022) introduces graph- based contrastive learning to preserve con- sistency regularization. Wang et al. (2021) incorpo- rates self-training into sequence labeling tasks by automatic weighting strategy.
Parameter-Efficien Learning. PEL is to opti- mize small portion of parameters while keep- ing the model backbone frozen, which aims at im- proving the training efficiency and preserving the model's effectiveness (He et al., 2022). Houlsby et al. (2019) integrates task-specific neural modules called adapters into PLMs, and only these adapters are updated during fine-tuning. Ptuning (Liu et al., 2021b) and Prefi--Tuning (Li and Liang, 2021) re- spectively introduce lightweight prefix module into the input layer and each transformer layer, abling efficient training over these prefix modules. Notable PEL-based models also include BitFit (Za- ken et al., 2022), LoRA, etc. This paper integrates PEL into self- traininiin to improve its efficiency.
We conduct extensive experiments over multiple NLU tasks. Results show that UPET outperforms strong baselines in terms of both effectiveness and efficiency. The improvement is consistent in dif- ferent settings with different PEL methods and the number of labeled data. Our key contributions to this field are summarized follows: 1) We use parameter-efficien learning of PLMs in conjunc- tion with uncertainty estimation to form an efficient and effective self Seff-training framework. 2) To better improve the robustness of the parfmeter-efficient model, we introduce Hard Contrastive Learn ing. 3) Extensive experiments among wide range of tasks demonstrate that our proposed framework outperforms prevailing strong baselines.
3 UPET: The Proposed Method
Given a labeled set Dj = unlabeled set D, = (X,t where N₁ and N, respectively denote the number of labeled set and unlabeled set (N; < N.). X,, X, E X' denote the input sentence in the labeled set and unlabeled set, respectively. Y, E ) is the corresponding label of X₃. The task is to train neural model f" and pseudo label for each unlabeled example X,, where f" + y is function with parameters W map the input space X' to the label space V. We aim to answer the following research problem:
and an
2 Related Work
Semi-supervised Learning and Self-training. SSL aims to effectively utilize unlabeled data in
How can we mitigate the problem of noisy pseudo labels via judiciously selecting
RQ1:

---

(1) Fine-Tuning
NES Closs
ACT POS
Closs
NEE
RO PDS
POS
Camoitty
Easy Exumples
t
Few-shot Labeled
X
x
POS
POS
ACC
NO POS
NEO POS
Exmple
Hard Exemples
NUM Hoad
Pseuda Annotation
Uncertain
Easy+Hord Contrastive Turing
cLu Houel
MLM Head
Pseudo
POS
Teacher
Parameter Efficient Turing
Student
(2) Reliable Example Sampling
(3) Efficient Robust Tuning
(4) Copy & Repeat Until Convergence
The overview of UPET framework. We first fine-tune teacher model over few-shot labeled data. Then, we aim to judiciously choose suitable pseudo-labeled data by uncertainty estimation. During student learning, we leverage the aarameter-efficiee method with robust PHCE loss and contrastive regularization to train the student model on secudo-labeled data. At last, the student model can be used for the next iteration. (Best viewed in color.)
Figure I:
reliable examples?
et al., 2022) to leverage information gain of the model parameters to show how certain the model is to the pseudo-labeled examples w.r.t the true labels 2 Typically, the information gain can be defined as:
RQ2: How the model parameters be efficiently updated during the iterative self- training process, meanwhile preserving the model's robustness and performance?
BY, D.) D.)
We thus propose the UPET framework which con- sists of two novel techniques, i.c., Reliable Exam- ple Sampling (RES) and Efficient Robust Tuning (ERT). The framework overview is illustrated in Figure I and the detailed algorithm is shown in Appendix B.
PxCW)p.)[HI(P)) W)].
(2)
where W denotes the parameters of the teacher. BY,, ,,,,, D.) denotes the information gain which the difference between HY,,,,,,, D.))
(the final entropy after seeing all examples from unlabeled sentences) and HYY,X,,, W) (the cur- rent entropy for the example X,). p(WID,) is the posterior distribution. As the calculation of Eq. is intractable, BNN to perform approximation. Specifically, we assume that the posterior distribution p( (D)) can be replaced with dropout distribution ge ( (W). Thus, we can sample T masked model weight (Wa) (W),,, and calculate the approxima- tion value as:
3.1 ThneTTunng and Pseudo Annotation
We start with a fine-tuning stage over the few-shot labeled data D, to form a teacher model fi After that, the pseudo label Y, of each unlabeled example X; can be generated by the teacher model:
utilize MC dropout in
Y, = arg max pax cl. (X)),,
(1)
where p(-) (.) is the probability distribution. However, the generated labels may be wrong due to the model confirmation bias problem. That means we need to explicitly reduce the noise problem by designing suitable sample selection strategy.
B BYY,WIX, = Σ
P.D log 1
Σnd
<ey
1=1
1 T
3.2 Reliable Example Sampling
Σ
Σ
S5. log(d:)
T
To reach this goal, follow Tsai et al. (2022); Mukherjee and Awadallah (2020); Hu and Khan
(-1
(3)
W. (X))) is the predict prob- where P'c = p(us = cl ability X, derived from the f-th masked model
(2021) to leverage uncertainty estimation from BNN measure what the reliable unlabeled ex- amples can selected for training. we fol- low (Houlsby et al., 2011; Gal et al., 2017; Tsai
"The model certainty can he used to estimate the reliability of the unlabeled example, even though the lahel is unknown.

---

Neg
(cis]
Head- Prad-Pttnning
(b) Head-Prefix
(c) Head-Adapter
21eat
Verbalizer
Verballeer
Verbatber
Powrive
Poerive
(d) Prompt-Ptuning
(c) Prompt-Prefix
(1) Prompe-Adapter
Figure 2: Overview different PEL paradigms. (a)-(c) represent Tennn, aiming to CLS head for prediction. (d)-(f) denote Prompt- Tuning make prediction via well-designed template and verbalizer. We unify three classic PEL methods for both Sead-TTunin and Prompt-Tuning. The block in light yellow and blue means the trainable and frozen parameters, respectively. The block with sketches denotes the adapter module. (Best viewed in color.)
W. Jieo Thus, lower B(Y,, WIX,, Da) value means that the model is more certain about the prediction, higher certainty corresponds lower informa- tion gain (Tsai al.. 2022)* Formally, we can design certainty score for each example as:
pseudo-labeled data. To meet our desiderata, we introduce two prediction paradigms with three PEL methods. The architecture is shown in Figure
Head- Tuning. Head- Tlniin leverages CLS head to generate the probability distribution of the given example. Formally, we have:
Head-Tuning.
= 1 B (Y, WIX,, D.).
(4)
Pw (,) (w
(6)
To this end, obtain the final sampling weight for each example by considering both model confi- dence and certainty:
where (-) denotes the output representation by the student model .uv"" Heta(-) denotes CLS head with a softmax classification layer" 4.
a S of +1 a)
=
(5)
of EXTED, ax 8 (1 - a.) x ss
Crompt-Tuning aims at reusing the Masked Language Modeling (MLM) head to make predictions. Specifically, a well-designed template T with masked token (" [MASK] is concatenated with the original input sentence. In addition, we need to define verbalizer that maps the probability distribution over the whole vocabu- lary set X' to the label set V. The probability can be calculated as:
Prompt-Tunine
where # tea (X))) is the model confidence derived from the average approx- imate posterior of the masked models w.r. the pseudo label Y, & (0 s a $ 1) denotes the bal- ancing factor. Hence, a number of N, reliable examples can be sampled by these weights to form new subset D, C D.,.
Pw* (x) (Mimm (TIIX,))) (7)
3.3 Efficient Robust Tuning
3.3.1
Parameter-Effiche Tuning
where Hymm denotes the MLM head derived from the PLM, -I1- is the concatenation operation. Vy(-) aims to map the label word's probability at the masked position to the corresponding class
After the annotation and selection of unlabeled ex- amples, we need to train student model to elicit knowledge from the teacher. Yet, the training pro- cess of the self-training paradigm is inefficient. To remedy this dilemma, we aim introduce PEL in self-training. We initialize a student model f and a few designated parameters in W* can be tuned, enabling efficiency when training on many
Hence, we can integrate Ptuning (Liu et al.. 2021b), Prefix-tuning (Li and Liang, 2021) and Adapter-tuning (Houlsby et al., 2019) to unify the PEL with arbitrary PLMs and prediction paradigms, including Peaad-tunning. Head Preffxx Hlcad-Adaple Prompt-Ptuning Prompt-Preffe and
'Intuitively, if the model is always certain about some examples, these examples might he too easy to contribute any additional information.
'I can be viewed as forwwrd network (FFN) with random initialized parameters.

---

Prompt-Adapter. More details are shown in Ap- pendix A.I. During the optimization, we can com- pute the following cross-entropy objective by:
term can be computed as:
Nr Σ Σ CE (N..)).D, Yice
1
W. 1 fotu
Σ (K.Y))ED,
log PBgPW-U2 Y,I
Nr
exp (g(X, X;
exp (() Σn exp (gl X
(8) Yet, it is still possible that the subset D, could con- sist of some wrong labels. During the parameter- efficient training stage, the scale of trainable pa- rameters in W* being small, the student model is fragile and the robustness could not be preserved due to the negative effect of these noises in the back- ward. In that, we follow (Tsai et al., 2022) to utilize partially huberised cross-entropy loss (PHCE loss), which is alternative variant with gradient clip- ping technique. Hence, the loss function in Liq. 8 can be modified as:
(10)
where ((:;)) is the score function that measures the similarity of two examples in the semantic
space. Finally, the whole training objective is de- signed as:
W'' C(Dr, W.: I(Dr, Jstu AR(S (11)
where À i the
Experiments
I(Dr,
W 1 N
0- = Yx (X,Y))DD
Σ
4.1 Dataset and Implementation Details
We perform extensive experiments over seven lan- guage understanding tasks to evaluate our UPET framework. We choose series of tasks from the GLUE benchmark (Wang et al., 2018), in- cluding SST-2 (Socher et al., 2013) for sentiment analysis, MNLI (Williams et al., 2018) 8) for lan- guage inference, QNLI (Rajpurkar et al., 2016) for question answering, MRPC (Dolan and Brock- ett, 2005) for semantic paraphrasing and RTE (Da- gan et al., 2005) for textual entailment. We also choose CB (De Marneffe et al., 2019) from Su- perGLUE (Wang et al., 2019) for linguistic en- tailment and AGNews (Zhang al., 2015) for topic classification. For each dataset, the num- ber of labeled examples per class is set as N₁ E {16, 32, 64). We repeatedly sample few-shot la- beled instances five times with different seeds from {12, 21, 42,87, 87, 100} and report average per- formance with standard deviation.
(9)
where (y(xx) is the PHCE loss function with hyper-pann appemmpareette (r 1). The detail of the PHCE loss function is shown in Appendix A.3.
3.3.2 Easy-Hard Contrastive Tuning
As mentioned above, the selected example in D, has a higher model certainty and might be too easy to contribute any additional information. Nonethe- less, this inevitably leads to the student model over-fitting on these frequently selected samples (Mukherjee and Awadallah, 2020). Intuitively, the example not selected in D, is more likely to be noise that results in semantic drift. Thus, natural idea is to exploit some hard examples (which are not selected in D,) as the negatives to keep them away from easy (reliable) examples, which can be viewed as a regularization in the semantic space.
To reach this goal, we present Easy-Hard Con- trastive Tuning. We denote Dn, as the difference between D, and D,,, so the examples in D,, repre- sent the hard ones. During the optimization of the student model, given one example (X,, Y) E D,,, we aim to choose one another example (X7, Y,+)
For the implementation details, we choose RoBERTa-aaree (Liu et al., 2019) from Hugging- Face as the default backbone for both the teacher and student model. The number of the self-training iterations is set as 5. We train models by the AdamW algorithm with B1 0.9, 32 0.98 on 4 NVIDIA V100-32G GPUs. For each task, we use grid search to select the best (Appendix D). By default, the training epoch of the teacher and student are 100.
from D, as the positive and some negative exam- ples ((Xi,, / from Dr, where N, is the number of negatives, Y, = Y Yú have the same class Hence, the contrastive regularization
'The pscudo lahel of the hard example may he wrong, so if the sampled hard example has the same label with ( (X,, Y.). it can be viewed as a negative in terms of the class Y..
Blltps://hugees i ing itg.ace.c///eooooo ctt.co///annteee index, html.

---

Baselines
Use PEL
#Tunahle Params.
SST-2 (acc)
MNLI (acc)
QNLI
MRPC (acc)
RTE (acc)
CR (acc)
AGNews (acc)
Avg.
(fl)
Full Data
Head FT
X
355M
95.2
89.8
93.3
91.4
83.0
90.5
94.7
91.1
Prompt FT
X
355M
95.9
90.2
93.0
90.9
88.4
91.1
94.0
91.9
Few Labeled Data (16-shot)
Head FT
X
355M
81.4338
45.826.4
60.226.5
75.9x2.9
54.4
74.5a26
88.9a2.7
68.7
Prompt FT
X
355M
90.6=1.1
53.7+2.3
64.5
74.4+3.0
59.1+3.6
77.0=3.3
88.6+1.2
72.6
Few Labeled Data (16-shot) Unlabeled Data
Head ST
X
355M
87 37.9
51.9=28
64,0+28
79.4+2.5
53.2+2.9
75.9=1.5
86,4+3..
71.2
Prompt ST
x
355M
91.0=3.1
57.7=29
17..2
81.0+2.
$7.943.3
77.7=29
88.8t3.5
UST
X
355M
84,0=4.0
53.9=2.
65.9t3.3
79.9+2.0
55.642.6
76.0a3.1
89.3+3.5
72.1
CEST
X
355M
86.4238
52.2s2.9
65.0a2.4
80.8
57.0+1.9
78.142.
88.5a2.2
LiST
14M
91.0=3.0
62.0s3.9
67.4
82.0g3.3
60.842.5
79.7429
90,342.5
76.2
UPET
Head-Ptuning
<IM
90.8=3.2
53.2+29
64, 8
82.6+2.8
59.3+3.7
76.8=26
90.8+1.8
74.0
Head-Prefh
87.5-20
56.7=2.7
69.2+3.1
82.3+2.2
58.7+25
79.621.5
90.9+1.8
Hcad-Adapter
14M
89.3=1.0
60.1
68.5&1.4
85.5+2.5
59.243.5
79.0s1.5
90,322
76.0
Prompt-Ptuning Ptuning
<IM
91.7=28
69.5s1.9
71.922.8
83.7
60.8+1.5
80.4+1.4
89.6a2.2
78.2
Prompt-Pree
<6M
92.3+2.0
64.2s2.9
66.1
83.0&1.8
61.5s1.6
80.82211
90,5a3.1
76.9
Prompt-Adee Adapter
14M
91.9-1.9
66. 66.1
66.8+1.8
84.2+1.4
61.0+1.6
80.4+2.0
91.0=2.0
77.3
Table I:
The performance comparison of accuracy or Fl scores (%) with standard deviations on seven tasks. All methods (except fine-tuning with full data) are trained with 16-shot labeled samples for each class and overall results are aggregated over five different runs with different random seeds. In UPET, the first three variants belong to the Head- Tunnng paradigm, while the others are Commpt-Tuning.
4.2 Baselines
tional few-shot learning (i.e., Head FT and Prompt FT). This demonstrates the impact self-training with unlabeled data. 2) We obtain the best overall performance of 78.2% with the lowest tunable pa- rameters (i.e., Prompt-PPuninng and improve over Head ST, Prompt ST, UST, CEST, and LiST by 7.0%, 3.6%, 6. %, 5.6%, and 2.0% respectively over seven tasks, which indicates that UPET out- performs state-of-the-aad in terms of both the ef- feetiveness and efficiency. 3) Compared to the strong baseline Prompt ST that uses the PEL-based approach, we obtain a 3.6% absolute improve- ment, demonstrating the substantial contributions of the well-designed reliable example selection and contrastive regularization. 4) We also list all 6 PEL paradigms' performance of UPET. We ob- serve that the performance of Prompt-Tuning is higher than Head-Tuning indicating that reusing the pre-training objective MLM with the task-orient template and verbalizer is more effective for self- training. In addition, more tunable parameters may enhance the student model's ability to learn seman- tic knowledge derived from the teacher.
We consider some strong baselines for comparison, including UST (Mukherjee and Awadallah, 2020), CEST (Tsai et al., 2022) and LiST (Wang et al., 2022). UST and CEST leverage uncertainty esti- mation for self-training. LiST integrates Adapter- tuning (Houlsby et al., 2019) into prompt-based learning for parameter-efffiienn self-training, which is similar to the Prompt-Adapter paradigm. In ad- dition, we also design two cemi-supereieed learn- ing baselines: 1) Head ST aims to use the classic fine-tuning with CLS head augment unlabeled data through standard self-training. 2) Prompt ST aims to reuse the MLM head with well-designed task-specific template and verbalizer to perform pseudo-labeling in standard self-training. We also choose Head FT and Prompt FT to fine-tune over few-shot or full training data.
4.3 Main Results
Table 1 illustrates the main results over seven NLU tasks with different settings. RoBERTa-large trained fully labeled examples provides the ceiling performance for the few-shot and semi- supervised setting. We thus make the following observations. 1) According to the overall results, all the methods with self- training outperform conven-
4.4 Further Analysis
Impact of Self-training Iterations. To validate the effectiveness self-training, we choose MNLI

---

Teacher
Student
Tunable
Avg.
Avg.
LIST
UPET
Use PEL
Use PEL
Params.
Result
Time
#-shot
16
32
64
16
32
64
Head-A Adapter
SST-2
91.0
91.8
92.7
91.9
93.0
93.6
355M+355M
76.6
11.3h
MNLI
62.0
65.7
69.7
66.1
69.2
72.3
/
355M+14M
76.0
4.1h
QNLI
67.4
71.5
74.4
71.1
75.0
x
14M+355M
75.2
10.7h
MRPC
82.0
84.2
85.8
84.2
85.1
85.7
14M+14M
75.0
3.8h
RTE
60.8
64.2
67.9
61.0
66.0
68.9
CB
79.7
83.1
85.7
80.4
84.3
86.2
Coomptt-Adaper
AGNews
90.3
90.8
91.3
91.0
91.4
91.9
x
355M+355M
77.6
11.0h
/
355M+14M
77.2
4.0h
Table 3: The performance (%) with different numbers (16/32/64 examples per class) of labeled data. The
X
14M+355M
76.4
10.7h
14M+14M
75.8
3.9h
parameter-effiiiet paradigm Prompt-Adapter.
Table 2: The average performance (%) over all tasks with different combinations of PEL paradigms.
forms LiST over most of the tasks no matter how many labeled training examples.
and RTE and draw some curves to show the perfor- mance of different PEL paradigms at each iteration in Figure 3.
Combination of Different Parameter-Efficien Learning Paradigms in Self-training. We aim to explore how PEL performs in the self-training procedure. We integrate the PEL paradigm into the teacher or student model to show the perfor- mance of the different combinations of PEL. As shown in Table 2, we choose Head-Adapter and Prompt-Adapter. We find the setting that all pa- rameters in both the teacher and student updated gains the best-average performance, indicating the ceiling performance of each paradigm. Yet, it costs about 11 hours which makes the self-training pro- cedure inefficient. addition, the time influence on whether the teacher model uses PEL is less than the student, because the teacher model only trains once while the student model needs to update for 100 epochs in each self-training iteration. Corre- spondingly, this motivated us to leverage PEL in the student model to improve the efficiency of self- training, preserving its effectiveness.
MNLI
RTE
70
65-
Accuracy (96)
60-
60-
55
55
50
Hodd Plams
45
50
40
35
45
Iteration
Iteration
Figure 3: The performance (%) of different self-training iterations over MNLI and RTE.
From the figure, we find that the performance increases when the framework continual training until the 4-th iteration, indicating the convergence our framework. Additionally, the student model with Prompt-Tuning (including Frompt-Ptuning Prompt-Prefic and Coompt-Adapter consistently outperforms Head- Head-Tuning Tuning (including Head-Tuning. Head- Prefix, and Tread-Adapted This shows that prompt-based methods can better utilize PEL to make self- fiff-traiiin both effective and efficient.
Selection Strategy
Avg. Results
None
76.0
a (w/o. Confidence)
77.2
Labeled Data Efficiency. To investigate the in- fluence of the number labeled examples, we vary the examples of each class from 16, 32, and 64.
a 0.2
77.9
a = 0.4
78.2
a = 0.6
77.6
We choose LiST as the strong baseline. To make fair comparison, the PEL select is Prompt- Adapter, which is the same LiST and only tunes the adapter module in PLM. Results in Table 3 il- lustrate that the performance gradually improves as the number labeled data increases, as expected. addition, we also find that our UPET outper-
a = 0.8
77.3
a = 1.0 (w/o. Certainty)
76.8
Table 4: The average performance (%) UPET
(Prompt-Ptuning) with different selection strategies (varying by a). "None" equals Prompt ST which trains student model on all pseudo-labeled data.

---

Methods
SST-2
MNLI
QNLI (fl)
MRPC
RTE
CB
AGNews (acc)
Avg.
(ace)
(acc)
(acc)
(acc)
(ace)
Prompt-Ptuning
Prompt ST
91.0
57.7
81.0
57.9
77.7
88.8
74.6
UPET
91.7
69.5
71.9
83.7
60.8
80.4
89.6
78.2
w/o. Reliable Example Sampling
91.3
63.0
69.8
82.2
58.3
78.3
89.2
76.0
w/o. certainty
91.4
65,8
70.4
82.8
59.0
78.6
89.5
76.8
w/o. confidence
91.6
66.3
83.3
59.7
78.8
89.5
77.2
w/o. PHCE loss
91.3
67.2
69.3
83.0
59.9
79.7
89.3
77.1
w/o. Easy-Hard Contrastive Tuning
91.5
65.8
68.5
82.8
58.9
79.1
89.6
76.6
Table 5: The 16-shot performance (%) of different variants of UPET with Prompt-Ptuning.
Methods
#Example
Accuracy
Visualization of the Contrastive Regularization. To investigate how the proposed Kasy-Hard Con- trastive Tuning contributes to the final performance, in Figure 4, we use the t-SNE (Van der Maaten and Hinton, 2008) tool and select the AGNews task for validation. Specifically, we randomly sample Ik testing examples to draw the representations in the semantic space. Results demonstrate that the model trained with contrastive regularization can make clearer boundary between every two classes, cor- roborating conclusions that avoiding the over- fitting problem and yielding better generalization.
Variational Pre-training
200
83.9
Reinforcement Adv. Training
100
81.7
SeqSSI+ + Self-training
100
78.5
SeqSSI.
100
76.2
SeqSSL Adv. Training
100
76.0
UPET (worst)
64
89.6
UPET (best)
64
91.0
Table 6: Performance comparison over AGNews task with non-BERT-bascd SSL approaches (Li and Ye, 2018; Gururangan et al., 2019; Dai and Le, 2015; Li and Sethy, 2020) (RL: Reinforcement Learning, Adv.: Adversarial, Temp. Ens.: Temporal Ensemble, Layer Part.: Layer Partitioning). UPET (worst) and UPET (best) denote the performance of Prompt-Ptuning and Prompt-Adapter.
4.5 Ablation Study
In this section, conduct ablation study demonstrate the impact of different variants of UPET that remove the designed technique. From Table 5, we thus make the following summarization. 1) We find that the performance of w/o. Reliable Example Sampling (RES) decreases a lot (more than 2%). In addition, we also find that the sam- pling weight considered by both certainty and con- fidence make consistent contributions in RES. These phenomena demonstrate the effectiveness of the de-noising approach considered by both model confidence and certainty. 2) Removing PHCE loss from UPET in 1.1 % performance drop in terms average results, which indicates the importance of PHCE loss in robust student training. 3) Through UPET versus UPET w/o. Easy-Hard Contrastive Tuning, the average performance of the student model is improved by about 1.6%, demonstrating the effectiveness of the contrastive regularization design.
Effectiveness of Reliable Example Sampling. To validate the effectiveness of the RES, we in- vestigate the effect of the balance factor in Eq. in terms of the average performance. From Table it is necessary to perform sample selection to tain more clean data. The results also illustrate that both model confidence and certainty substantially make contribute to the performance. We find the best value is set around 0.2, which means certainty plays important role in the selection.
Nordd
Vorld
toors
4.6 Comparison to Non-BERT Approaches
We end this section with an additional comparison between UPET and non- BERT semi-supervised learning approaches that use a different number
Figure 4: The AGNews's t-SNE visualization of UPET w/o. Easy-Hard Contrastive Tuning (left) and w/ Easy- Hard Contrastive Tuning (right).

---

of labeled examples for tuning the teacher model. Table 6 shows that our framework achieves large performance gain with only 64 labeled examples, especially on UPET (best) with at least 7%.
References
Massih-Reza Amini, Vasilii Feofanov, Loic Pauletto, Emilie Devijver, and Yury Maximov. 2022. Self- training: A survey. CoRR, abs/2202. 12040.
Nitesh V. Chawla and Grigoris L. Karakoulas. 2005. Learning from labeled and unlabeled data: An em- pirical study across techniques and domains. JAIS, 23:331-366.
5 Conclusion
In this paper, we introduce a novel uncertainty- aware parameter-efficien self-training framework (UPET) to better improve the effectiveness and efficiency of self-training. In UPET, we use certainty estimation to judiciously select reliable pseudo-labeled examples to explicitly alleviate the noisy label problem. To make self-training more efficient, we integrate multiple parameter-efficient paradigms into self-training. To further improve the performance, we also present Easy-Hard Con- trastive Tuning to enhance the robustness and re- duce the over-fitting problem. In the future, will extend our framework to other complex tasks, such as sequence labeling, question answering, etc.
Jiaao Chen, Zichao Yang, and Diyi Yang. 2020. Mixtext: Linguistiaall-informie interpolation of hidden space for seni-supervised text classification. In ACL, pages 2147-2157.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The PASCAL recognising textual entailment challenge. In MLCW, volume 3944 of Lecture Notes in Computer Science, pages 177-190. Springer.
Andrew M. Dai and Quoc V. Le. 2015. Semi-supervised sequence learning. In NIPS, pages 3079-3087.
Marie-Catherine De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmenthank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Limitations
Our limitations are shown below:
Jacob Devlin, Ming-Was Wei Chang. Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In NAACL, pages 4171-4186.
We only focus on sequence -lassification-style NLU tasks. However, we think it can be ex- tended to other tasks easily, such as sequence labeling, question answering, etc.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zong- han Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Tai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. 2023. srrameter--fficiee fine-tuning of large- scale pre-trained language models. Nature Machine Intelligence, 5(3)
Our work focuses on the PLM without Trans- former decoders. We think extend our method to natural language gen- eration (NLG) tasks. We will leave as our future work.
possible to
Ethical Considerations
William Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In /WP@ IJCNPP. Asian Federation of Natural Lan- guage Processing.
Our contribution in this work is fully methodologi- cal, namely uncertainty- parameter- efficcttt self-training (UPET) to improve effectiveness and efficiency based on PLMs. However, transformer- based models may have some negative impacts, such as gender and social bias. Our work would unavoidably suffer from these issues. We suggest that users should carefully address potential risks when the UPET models are deployed online.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML,, volume 48, pages 1050-1059.
Yarin Gal. Riashat Islam, and Zoubin Ghahramani. 2017. Deep bayesian active learning with image data. In ICMI., volume 70, pages 1183-1192.
Suchin Gururangan. Tam Dang, Dallas Card, and Noah A. Smith. 2019. Variational pretraining for semi-supervised text classification. In ACL, pages 5880-5894.
Acknowledgements
This work has been supported by the National Natural Science Foundation of China under Grant No. U191 1203, and the National Natural Science Foundation of China under Grant No. 6237227
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig, 2022. Towards unified view of parameter-efficient transfer learning. In ICLR.

---

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In ICML. volume 97 of Proceedings of Machine Learn- ing Research, pages 2790-2799.
Guo-Jun Qi and Jiebo Luo. 2022. Small data challenges in big data era: A survey of recent progress on unsu- pervised and semi-supervised methods. IEFE Trans. Pattern Anal. Mach. Intell., 44(4):2168-2188)
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Neil Houlsby. Ferene Huszar, Zoubin Ghahramani, and Máté Lengyel. 2011. Bayesian active learning for classification and preference learning. CoRR,
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng, and Christopher Potts. 2013. Recursive deep mod- els for semantic compositionality over sentiment treebank. EMNLP, pages 1631-1642. ACL.
abs/1112.5745.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. 2022. Lora: Low-rank adaptation of large language models. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky, Πlya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. JMLR, 15(1):1929-1955
Yibo Hu and Latifur Khan. 2021. Uncertainty-aware reliable text classification. In KDD, pages 628-636. ACM.
Austin Cheng- Yun Tsai, Sheng-Ya Ya Lin, and Li-Chen Fu. 2022. Contrast-enhanced semi-supervised text classification with few labels. In AAAI, pages 11394- 11402.
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. CoRR, abs/2205.11916.
Laurens Van der Maaten and Geoffrey Hinton, 2008 Visualizing data using t-sne. Journal of machine learning research, 9(11).
Alexander Hanbo Li and Abhinav Sethy. 2020. Semi- supervised learning for text classification by layer partitioning. In ICASSP, pages 6164-6168. IREE,
Jesper E. van Engelen and Holger H. Hoos. 2020. A survey semi-supervised learning. Mach. Learn., 109(():373 440.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL/IJCNLP, pages 4582-4597, Association for Computational Linguistics.
Alex Wang, Yada Pruksachatkun. Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel R. Bowman. 2019. Superglue: stickier benchmark for general-purpose language understand- ing systems. In NeurIPS, pages 3261-3275.
Yan Li and Jieping Ye. 2018. Learning adversarial net-
works for semi-supervised text classification via pol- icy gradient. In SIGKDD, pages 1715-1723, ACM.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. GLUE: A multi-task benchmark and analysis plat- form for natural language understanding. In EMNLP. pages 353-355.
Xiao Liu, Kaixuan Ji. Yicheng Fu, Zhengxiao Du. Zhilin Yang, and Jie Tang. 2021a. la P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/2110.07666
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. GPT understands, too. CoRR, abs/2 103.55
Hao Wang and Dit- Yan Yeung. 2016. Towards bayesian deep learning: A framework and some existing meth- ods. IEEE TKDE, 28(12)3395-3448.
Yinhan Liu, Myle Ott. Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1 1162..
Yaqing Wang, Subhabrata Mukherjee, Haoda Chu, Yuancheng Tu, Ming Wu, Jing Gao, and Ahmed Has- san Awadallah. 202 1. Meta self-training for few-shot neural sequence labeling. In KDD, pages 1737-1747.
Yuning Mao. Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. Unipelt: A unified framework for parameter-efficien language model tuning. ACL, pages 0253-6264 Association for Computational Linguistics.
Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao. Ahmed Hassan Awadallah, and Jianfeng Gao. 2022. List: Lite prompted self self-training makes parameter-effiiiet few-shot learners. In NAACL, pages 2262-2281.
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2018. broad-coverage challenge corpus for sentence understanding through inference. NAACL-HLLT, pages 1122-122..
Subhabrata Mukherjee and Ahmed Hassan Awadallah. 2020. Uncertainty-awee aware self self-training for few-shot text classification. In NewrIPS.

---

Qizhe Xie. Zihang Dai, Eduard H. Hovy, Thang Luong, and Quoe Le. 2020. Unsupervised data augmentation for consistency training. In NeurIPS.
Adapter. Adapter-tuning (Houlsby al., 2019) designs multiple adapter networks into the trans- former bloc, which can be viewed as two feed- forward projections. Specifically, the adapters first project the original d-dimensional features into a smaller dimension, m, apply a nonlinearity, and then project back to d dimensions. The total num- ber of parameters added per layer, including biases, is 2md + d m. By setting < d, we limit the number of parameters added per task.
Xiangli Yang. Zixing Song, Irwin King, and Zenglin Xu. 2021a. A survey deep semi-supervised learning. CoRR, abx/2103.00550)
Xiangli Yang, Zixing Song, Irwin King, and Zenglin Xu. 2021b. A survey on deep semi-supervised learning. CoRR, abs/2103. 00500000
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quoc 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. In NeurlPS, pages 5754-5764.
We extend these methods into two paradigms, i.e. Head-Tuning Tuning and Prompt-Tunie
Head-TTnnnng aims to stack the prediction layer based on the CLS head, while Prompt- Tuning aims to reuse the pre- training objective of Masked Language Modeling (MLM) and predict by the well-designed template and verbalizer. As shown in Figure 2, we can unify all parameter-efficien methods with both Head- Tuning and Prompt-Tuning.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg- Kerg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convo- lutions. In ICML, volume 70, pages 3881-3890.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer-base masked magguage-model, In ACI, pages 1-9. Association for Computational Lin- guistics.
A.2 Bayesian neural network (BNN)
We provide brief introduction to BNN (Mukher- jee and Awadallah, 2020). Given neural model fW and training set Dr, the parameters W can be optimized by the posterior distribution (W)).) In the inference stage, suppose that we aim to gen- erate the label for the unlabeled example X; E D.. we can calculate the probability distribution by:
Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In NeurlPS, pages 649-657.
Zhenru Zhang, Chuanqi Tan, Haiyang Xu, Chengyu Wang, Jun Huang, and Songfang Huang. 2023, To- wards adaptive prefix tuning for parameter-efficient language model fine-tuning. ArXiv, abs/2305. 15212.
A Background Knowledge
p(y = c|x,) = =s ply= cIS (X))((1)D.)))
A.1 Parameter-Efficien Learning Paradigms
(12)
PEL aims to update the partial parameters of the PLMM to improve the training efficiency (Mao et al., 2022). We first introduce three classic parameter- efficient methods.
In other words, BNN averages over all the possi- ble weights instead of directly optimizing for the weights (Mukherjee and Awadallah, 2020). Yet, it is intractable in practice for Eq. 12, that find a surrogate distribution make the calculation tractable. Specificallyy we consider go (W) to be the dropout distribution (Srivastava et al., 2014) which aims to sample T masked model weights from the current model. Hence, the approximate posterior for each unlabeled example is:
Ptuning. Ptuning (Liu al., 2021b) adds con- tinuous prompt into the input and uses a prompt encoder to realize parameterization. Specifically, for each input sequence X, we have a task-specific prompt template T as follows:
"I, Pj, Xt was MASK.
1 T E cl (13)
where P. is prompt pseudo token (as proposed in Liu et al. (2021b)), is the total number of pseudo tokens, and MASK is a special token as the placcholder for model output.
pvy = clx,)
T
where (Wite1 (W) are the masked model weights.
Prefix-tuning. Prefix-tuning (Li and Liang, 2021) (it can also be viewed as P-tuning V2 (Liu al.. 2021a)) extends the key and value matrix with new continuous vectors in each transformer layer. We also denote the length of the prefix vectors as I.
A.3 Partially Huberised Cross-Entropy
Partially huberised cross cross-entropy loss (PHCE loss) can be used to alleviate the noisy label problem

---

Category
Dataset
#Class
#Train
#Test
Type
Labels (classification tasks)
SST-2
6,920
872
sentiment
positive, negative equivalent
MRPC
2
3,668
408
paraphrase
equivalent, not eeeeeoooettttteee entailment, neutral, contradiction entailment, e__eettilment entailment, not_ cot_ammmlm entailment, neutral, contradiction world, sports, business, technology
Text
MNLI
3
392,702
9,815
NLI
QNLI
2
104,743
5,463
NLI
Classification
RTE
2
2,490
277
NLI
CB
3
250
57
NLI
AGNews
4
120,000
7,600
topic cls.
Table 7: The statisties of multiple languages understanding tasks. Since the original test data is unavailable, use the development sets as our test sets.
via simple variant of gradient clipping for the classification loss (e.g. cross-entropy). Given one example (z, y), the PHCE loss o(x, y) is denoted as:
Teacher Hyper-parameter
Value
Batch Size Seed
(12. 21, 42, 87, 100)
Examples per Class
[16, 32, 64)
[0.1, 0.3, 0.5, 0.7, 0.9)
[0.001, 0,01, 0.05, 0.1, 0.5, 10)
TPw y) log T +1 pw (w( 1 1/T: log v) (2, 8) > 1/T:
Prefix Length
[4, 8, 16, 32, 64, 128)
Adapter Small Dim
[8, 16, 32, 64, 128, 256]
(14) where > the hyper-parameter. Thus, the model learned by Eq. 14 can be more robust to the noisy labeled tokens than the common cross- entropy.
Table 8: The searching scope for each hyper-parameter.
D Searching Scope of Grid Search
We use grid search to select the best hyper- parameters for each task, the searching score is shown in Table 8.
B Self-training Procedure
We show the whole training procedure in Algo- rithm 1. Specifically, we first use the original PLM
Algorithm Self-training Procedure of UPET
Require:
Neural model f labeled data D,, unlaheled data D.,-
w to initialize a teacher model Ji (Algorithm 1, Line I), and then fine-tune the teacher model over few-shot labeled data Dj (Algorithm 1, Line 2). During the iteration process, we sample a subset unlabeled set D', from Du, and obtain model con- fidence and certainty for each unlabeled example X (Algorithm Line 4). Based these factors, we can calculate the sampling weight for each unla- beled example and sample some reliable examples to form an casy set D,, and the rest is formed as a hard set DA (Algorithm 1, Line 7-9). During the student learning, we use the original PLM (Wo to initialize a student model por and use PHCE loss and Easy-Hard Contrastive Tuning to train the parameter-efficient student model over the pseudo- labeled examples (Algorithm 1, Line 6, 10-12).
1: Initialize a teacher model fiw, = f"o;
2: Fine-tune the teacher model from over the laheled data D,
(All parameters will be updated);
3: while not converged do
Sample an unlabeled data subset D', c D.:
S: Pseudo annotate each unlabeled example X, D', by feem in Eq. to ohtain the hard label Y;:
6:
Initialize student model f = pWo. Obtain the certainty score for X,:
7:
8: Obtain the confidence score s for X.: Sample reliable examples to form a subset D, by the sampling weight in Eq. 5. The examples not sampled can be used to form 2),-
9:
10:
Calculate the PHCE loss I(D,,. _W',, in Eq. 9; Calculate the regularization loss RCS in Eq. 10: Training fur. via parameter-efficient learning by re- duce ((", in Eq. 11; Update the teacher model W f
11:
12:
13:
At last, can copy the parameter of the student model the teacher and repeat until convergence.
14: end while
15: return The teacher model fre-
C Details of NLU task
We list the statistics each task in Table


Reviews:
[{"review_id": "inu4KmE42N", "reviewer": "Reviewer_a1vW", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are very grateful for receiving your review, and here are our responses:\n\n> The authors state in the limitation section that they focused solely on pre-trained language models that do not use transformer-based decoders. It would be nice to see results from this side of the 'transformer family tree', but not a show-stopper given the thorough experiments described in this work.\n\nThank you for reviewing our limitations sections. The pseudo-labeling framework with uncertainty estimation needs to obtain the logits (probability distribution) for T times. It will take too much time when apply for generative LLM. However, if we can obtain the logits for each generated token, the proposed UPET can be easily adapted to generative LLM. We will follow your suggestions and are planning to extend the UPET to more backbones in the future, such as conducting additional experiments using models based on the T5 architecture.\n\n> Typos Grammar Style And Presentation Improvements\n\nWe deeply appreciate your meticulous review of our paper and for identifying this issue. Indeed, it's a typo, and we have made the corrections. Thank you again for helping improve our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors walk through their approaches with clear explanations of the various design choices and mathematical motivations. Experiments are performed against a series of strong baselines and a thorough combination of their approaches to demonstrate gains in F1 scores on a number of natural language understanding test sets and take care to run these experiments a number of times to report statistically significant results.", "reasons_to_reject": "The authors state in the limitation section that they focused solely on pretrained language models that do not use transformer-based decoders. It would be nice to see results from this side of the 'transformer family tree', but not a show-stopper given the thorough experiments described in this work.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors present an adaptation of the teacher-student paradigm to allow training on massive amounts of unlabelled data in a semi-supervised fashion. This work specifically focuses on natural language understanding (NLU) but they posit their approach could be used for a number of other tasks. The framework presented in this work uses Monte Carlo sampling and Bayesian neural networks in the teacher network to sample high-confidence examples to use in training a student network. Varying the difficulty of examples presented to the student network acts as a regularizer effect to ensure that the student does not overfit on shorter, easier training sequences.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Tables 1 and 2 have a column for '# Tuable Params' - is this meant to be Tunable Params?"}, "extra_text": {}}, {"review_id": "j44lFatDQf", "reviewer": "Reviewer_exqi", "summary": "", "questions": "- Is PEL the same as PEFT? PEFT is widely accepted in the community IMO. So, you may want to stick with the PEFT term.\n- How are proposed ERT methods different from PEFT methods? Are they not the same with the only difference that now your dataset is obtained in SSL setup instead of a manually labeled dataset?\n- Did you try to look at the papers on hard example mining for contrastive learning? How does your method compare to other approaches, e.g., the active bias approach or hard mining example approach from the vision domain? \n- What does improvement stops after a few iterations? Is it something observed in prior works on self-training as well?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for carefully reviewing our paper, we hope our responses can address your concerns.\n\n> Q1: Is PEL the same as PEFT? PEFT is widely accepted in the community IMO. So, you may want to stick with the PEFT term.\n\nThanks for your query about the terms. PEL is parameter-efficient learning which is the same as PEFT (parameter-efficient fine-tuning). Our proposed module PET belongs to PEL, however, it still has some differences with PEFT: 1) We propose two prediction paradigms, i.e.,  head-tuning (add a new MLP head for cls) and prompt-tuning (add template and verbalizer to reuse MLM head), 2) Our PET can integrate some PEL methods (e.g., Adapters, LoRA from PEFT toolkit) with above tow difference paradigms. For example, we can leverage templates and verbalizers to perform prompting with only tuning adapter modules.\n\n> Q2: How are proposed ERT methods different from PEFT methods? Are they not the same with the only difference that now your dataset is obtained in SSL setup instead of a manually labeled dataset?\n\nThanks for this question. Our proposed ERT module leverages PEFT to make the training processing more efficient. The main differences are: 1) ERT consists of two main stages, such as parameter-efficient tuning on two prediction paradigms (i.g., Head-tuning and Prompt-tuning); 2) ERT also improves model robustness when training on pseudo-labeled data by using contrastive learning.\n\n> Q3: Did you try to look at the papers on hard example mining for contrastive learning? How does your method compare to other approaches, e.g., the active bias approach or hard mining example approach from the vision domain?\n\nWe do not compare other hard example approaches because that the hard sample weight derived from uncertainty estimation can make the framework end-to-end. We do not focus only on hard examples of mining in this work. However, we will add these methods as our new baselines and perform external experiments in the revised version.\n\n> Q4: What does improvement stop after a few iterations? Is it something observed in prior works on self-training as well?\n\nThanks for your astute observations. The improvement stops when training after about 4 iterations, which suggests that our framework can converge quickly. As shown in Figure 3, especially, when training after one iteration, more reliable unlabeled examples may selected for training the student model. When training more than about 4 iterations, there are fewer unlabeled data that may be useful for improvement. Sometimes, when the iteration number is larger, the model may overfit and make more wrong predictions.\n\n> About Missing References\n\nWe sincerely thank you for pointing out these two important papers that are worth citing. We will incorporate them into the related works section in the final version or directly discuss their relevance.\n\n> About Typos Grammar Style And Presentation Improvements:\n\nWe sincerely appreciate your attention to detail in pointing out the typos in the notations, as well as alerting us to the citation format errors in Lines 195 and 196. We will make the necessary corrections in the final version. Additionally, we aim to further refine the notations to make the paper more comprehensible. Regarding your feedback on the BNN section, we wholeheartedly agree and will revise this section for clarity.\n\n> Responses to Reasons To Reject\n\nMoreover, we have carefully read your comments in the \"reasons to reject.\" For issues regarding PEL / ERT steps, please refer to our response in Q1. We also appreciate your attention to tasks beyond natural language understanding. We concur and plan to expand our experiments to encompass other NLP tasks such as question answering, text generation, and translation. Specifically, in the final version, we aim to include more tasks like SQuAD as further evidence of UPET's effectiveness.\n\nRegarding your concerns about Easy-hard contrastive learning, under the context of self-training, it is an original idea from us. The setting for t-SNE plots is designed for readers to intuitively grasp our approach. Moreover, we fully acknowledge and support your recommendation on contextualizing prior work. In the upcoming version updates, we will add additional previous works to better position our study within the existing literature.\n\nThanks again for your careful reviewing and insights!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper addresses a very important SSL problem of selecting samples using uncertainty derived from BNNs. While BNNs are well known, this paper presents experiments and results how they can be used for this particular problem in the self-training setup. \n- Empirical results are impressive, e.g., Table 1 shows substantive improvements over other baselines.", "reasons_to_reject": "- The methodological innovations behind PEL / ERT steps seem similar to the PEFT setup, and not clear if it adds any new too or insights for NLP practitioners/researchers.  \n- The method has many components which makes it harder to gauge the crucial part and relative importance of all the components, especially when practitioners need to apply these methods to novel scenarios, i.e., scenarios not considered in the paper. \n- Easy-hard contrastive learning is presented without contextualizing prior work in this area. It’s unclear how and why this method is better than prior works. Empirical evidence to support this part of the proposed work is also not satisfactory as t-SNE plots can be easily misleading.", "questions_to_authors": "- Is PEL the same as PEFT? PEFT is widely accepted in the community IMO. So, you may want to stick with the PEFT term.\n- How are proposed ERT methods different from PEFT methods? Are they not the same with the only difference that now your dataset is obtained in SSL setup instead of a manually labeled dataset?\n- Did you try to look at the papers on hard example mining for contrastive learning? How does your method compare to other approaches, e.g., the active bias approach or hard mining example approach from the vision domain? \n- What does improvement stops after a few iterations? Is it something observed in prior works on self-training as well?", "paper_topic_and_main_contributions": "This paper proposes a new self-training method that uses an uncertainty-aware machine learning model, specifically Bayesian neural networks (BNNs). The method first exploits the model's confidence and uncertainty to select reliable samples from unlabeled data. These samples are then used to perform PEFT-style fine-tuning on the student network. The paper also proposes an easy-hard contrastive tuning method to further improve the performance of the student network. Towards the end, this work presents strong empirical results validating the efficacy of the proposed methods.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Following two papers might be worth citing: \n\n- Active bias paper – https://arxiv.org/abs/1704.07433 – Haw-Shiuan Chang, Erik Learned-Miller, Andrew McCallum, Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples, NeurIPS 2017. \n- Hard example mining – A. Shrivastava, A. Gupta, and R. Girshick. Training region-based object detectors with online hard example mining. In CVPR, 2016", "Typos_Grammar_Style_and_Presentation_Improvements": "- Line 189: p is the probability distribution but the domain of the distribution is not mentioned clearly. ‘c’ variable is not defined as well. May be you can omit ‘c’ altogether. \n- Line 195 and 196: cite in text without brackets. \n- Notations are confusing in the methodological section, e.g., sometimes W has tilde over it in line 217 and other times it is missing. \n- Writing for the BNN section is unclear and requires substantive gazing efforts. Also, it seems possible to cut few details that are not required to sub-select samples."}}, {"review_id": "4OmZmJRC33", "reviewer": "Reviewer_rnmD", "summary": "", "questions": " I will reconsider my score in the rebuttal.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We truly thank you for your constructive criticism and valuable suggestions as to how to make this paper stronger.\n\n> R1: While the paper does an excellent job of demonstrating the effectiveness of UPET in low-resource scenarios, it could benefit from more discussion and testing in high-resource scenarios. Understanding how UPET performs when abundant labeled data is available would provide a fuller picture of its versatility and performance.\n\nThank you for your suggestions. In this work, we mainly focus on low-resource scenarios, which are more challenging than high-resource scenarios. If the labeled data is abundant, we think the model can better capture semantics and there is no need to perform semi-supervised learning. To this end, we will conduct additional experiments using more labeled data / more shots and analyzing UPET's performance under identical settings.\n\n> R2: The paper could further discuss and explore the choice of parameters that are updated during the efficient training method. A more detailed analysis on this aspect could provide insights into why certain parameters are more beneficial to update and how it affects the overall performance.\n\nThank you very much for your suggestion regarding the choice of parameters that are updated. Due to space constraints, we primarily focus our analysis on self-training and label efficiency to demonstrate the rationale behind our main framework. Following your advice, we will add a new set of experiments (along with statistics) to the appendices (or the additional one page in the camera-ready version) in our next revision, analyzing why certain parameters are more beneficial to update and how they will impact overall performance.\n\n> R3: The paper could also benefit from comparisons with other semi-supervised learning methods or other parameter-efficient training methods. This would help to position UPET within the broader field of semi-supervised learning for NLP.\n\nWe have conducted some experiments as the reviewer mentioned. For the comparison with other semi-supervised learning methods, the results in Table 6 show that our method can achieve the best performance. For the different PEL methods, we have conducted the comparison on six different PEL settings in Table 1. The results demonstrate that P-tuning with the Prompt paradigm can outperform other PEL methods.\n\n> R4: While the paper does present results on several tasks, including more diverse tasks or languages could further demonstrate the robustness and generalizability of UPET.\n\nThank you for your suggestion. We will conduct experiments over other NLP tasks, such as question answering, text generation, and translation. Specifically, in the final version, we will incorporate tasks like SQuAD as further evidence of UPET's effectiveness.\n\nWe sincerely appreciate your insightful and valuable comments/criticisms, and we are very grateful for your willingness to reconsider the score in the rebuttal. Thank you for your suggestions that can make this paper stronger.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper addresses a significant problem in the field of NLP, i.e., the dependence of large pre-trained language models on extensive labeled data, which is often not available in low-resource scenarios.\n2. The proposed UPET model innovatively incorporates uncertainty estimation and an efficient parameter-updating method into the self-training loop, leading to better handling of noise and computational efficiency.\n3. The paper provides extensive experimental results, showcasing the effectiveness of UPET in various language understanding tasks and in different resource settings. It demonstrates that UPET can outperform baseline methods, indicating its practical utility.\n4. The methodology and experimental setup sections of the paper are detailed and clear, making it possible for other researchers to reproduce the results.\n", "reasons_to_reject": "1. While the paper does an excellent job of demonstrating the effectiveness of UPET in low-resource scenarios, it could benefit from more discussion and testing in high-resource scenarios. Understanding how UPET performs when abundant labeled data is available would provide a fuller picture of its versatility and performance.\n2. The paper could further discuss and explore the choice of parameters that are updated during the efficient training method. A more detailed analysis on this aspect could provide insights into why certain parameters are more beneficial to update and how it affects the overall performance.\n3. The paper could also benefit from comparisons with other semi-supervised learning methods or other parameter-efficient training methods. This would help to position UPET within the broader field of semi-supervised learning for NLP.\n4. While the paper does present results on several tasks, including more diverse tasks or languages could further demonstrate the robustness and generalizability of UPET.", "questions_to_authors": " I will reconsider my score in the rebuttal.", "paper_topic_and_main_contributions": "1. The development of UPET, an uncertainty-aware and parameter-efficient self-training framework.\n2. Incorporation of Monte Carlo dropout in the self-training loop to estimate model uncertainty.\n3. Introduction of an uncertainty-aware pseudo-labeling mechanism to reduce noise in the pseudo-labeled data.\n4. Introduction of a parameter-efficient training method to update only a small fraction of the model parameters.\n5. Empirical demonstration of UPET's effectiveness in low-resource scenarios across various language understanding tasks.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work presents develops an uncertainty-aware and parameter-efficient self-training framework called UPET, which involves using uncertainty estimates to select reliable samples from unlabeled data and introduces an easy-hard contrastive tuning method. The experiments are conducted against a series of baselines and provide statistically significant results that demonstrate gains in F1 scores on a number of natural language understanding test sets. In addition to the positive comments, the reviewers also present some concerns: Reviewer 1 notes that the authors focus solely on pretrained language models that do not use transformer-based decoders,  Reviewer 2 notes that the methodological innovations behind the proposed methods may not add significant new insights or techniques for NLP practitioners and researchers, and Reviewer 3 suggests that the paper could benefit from more discussion and testing in high-resource scenarios, and a more detailed analysis of the choice of parameters that are updated during the efficient training method.
Decision: Accept-Findings