SlowGuess/NAACL_2022_OCR huggingface dataset:
Labels	Tuning	#Params	Avg	MNLI (m/mm) (acc)	RTE (acc)	QQP (acc)	SST-2 (acc)	Subj (acc)	MPQA (acc)
|K|=10	Full	355M	69.3	54.8 (3.7) / 55.6 (4.6)	60.0 (4.4)	58.7 (4.6)	89.5 (1.7)	84.5 (8.6)	67.8 (6.9)
Embedding	53M	62.3	53.3 (1.1) / 53.7 (1.2)	56.1 (3.5)	50.9 (6.4)	84.4 (3.6)	70.3 (6.0)	58.8 (7.0)
Attention	101M	68.0	55.1 (3.0) / 55.8 (4.0)	57.9 (3.9)	57.8 (7.0)	90.3 (1.5)	82.0 (6.6)	64.3 (6.6)
FF-output	102M	69.0	55.7 (3.3) / 56.4 (4.0)	60.4 (4.3)	59.1 (5.7)	90.2 (1.5)	82.2 (7.1)	66.2 (8.1)
FF-intermediate	102M	67.1	55.0 (2.8) / 55.7 (3.7)	57.7 (3.5)	57.0 (7.2)	89.3 (2.1)	80.7 (6.1)	62.7 (6.9)
|K|=20	Full	355M	75.4	60.3 (2.0) / 61.6 (2.7)	64.3 (2.4)	67.8 (4.2)	90.6 (1.8)	88.3 (2.2)	80.6 (7.5)
Embedding	53M	65.6	53.2 (1.3) / 53.1 (1.5)	58.1 (0.9)	55.7 (5.2)	86.0 (1.7)	78.0 (2.0)	62.7 (3.2)
Attention	101M	74.6	59.2 (1.7) / 60.2 (2.4)	61.4 (2.2)	66.8 (2.6)	91.7 (1.1)	88.6 (1.5)	79.3 (5.5)
FF-output	102M	75.7	60.2 (1.8) / 61.4 (2.6)	65.2 (2.5)	67.7 (3.4)	91.4 (1.4)	88.5 (1.3)	80.3 (5.2)
FF-intermediate	102M	73.5	58.3 (1.6) / 59.3 (2.0)	60.8 (2.3)	66.2 (3.2)	90.5 (1.3)	87.4 (2.3)	77.4 (5.8)
|K|=30	Full	355M	77.6	62.8 (2.6) / 64.1 (3.3)	66.1 (2.2)	71.1 (1.5)	91.5 (1.0)	91.0 (0.5)	82.7 (3.8)
Embedding	53M	67.0	54.1 (1.1) / 54.0 (1.2)	59.0 (2.7)	56.7 (4.5)	85.8 (0.9)	82.2 (2.6)	64.2 (2.1)
Attention	101M	77.0	61.6 (2.2) / 62.7 (2.9)	65.8 (3.2)	70.1 (2.2)	91.7 (0.9)	90.4 (0.7)	82.1 (2.5)
FF-output	102M	77.6	62.3 (2.1) / 63.5 (3.0)	67.3 (2.6)	70.8 (1.7)	91.8 (0.8)	90.2 (1.3)	82.5 (3.4)
FF-intermediate	102M	75.9	60.4 (1.9) / 61.4 (2.5)	64.0 (3.9)	69.0 (2.7)	91.0 (1.2)	90.0 (1.3)	80.7 (2.7)
Table 11: Average performance and standard deviation on tuning different modules of RoBERTa-large with varying amount of training labels  $|\mathcal{K}|$ .
Tuning	#Params	MNLI (m/mm)	RTE	QQP	SST-2	Subj	MPQA
Head-only	1M	54.1 (1.1) / 54.1 (1.3)	58.8 (2.6)	56.7 (4.5)	85.6 (1.0)	82.1 (2.5)	64.1 (2.1)
Bias-only	1M	54.4 (1.3) / 54.4 (1.5)	59.8 (3.5)	58.6 (4.4)	87.3 (1.1)	83.9 (2.3)	65.8 (1.8)
Prompt-only	1M	47.3 (0.2) / 47.7 (0.1)	53.0 (0.6)	39.9 (0.7)	75.7 (1.7)	51.5 (1.4)	70.9 (2.4)
LiST Adapter (2)	1M	56.3 (3.8) / 57.1 (4.7)	63.7 (4.9)	68.2 (2.4)	89.2 (0.9)	90.2 (0.8)	68.4 (3.0)
Houlsby Adapter	14M	35.7 (1.1) / 36.2 (2.0)	51.0 (3.0)	62.8 (3.0)	57.0 (6.2)	83.2 (5.4)	57.2 (3.5)
LiST Adapter (128)	14M	62.4 (1.7) / 63.7 (2.5)	66.6 (3.9)	71.2 (2.6)	91.7 (1.0)	90.9 (1.3)	82.6 (2.0)
Full tuning	355M	62.8 (2.6) / 64.1 (3.3)	66.1 (2.2)	71.1 (1.5)	91.5 (1.0)	91.0 (0.5)	82.7 (3.8)
Table 12: Average performance and standard deviation of several lightweight parameter-efficient prompt-tuning strategies with  $|\mathcal{K}| = 30$  training labels. The best performance is shown in **bold** along with the number (#) of adapter parameters of total encoder parameters.
Backbone	Approach	Average Acc
BERT-base	Prompt FN	66.0
BERT-base	MetaST	60.2
BERT-base	PromptST	66.1
BERT-base	LiST	68.6
BERT-large	Prompt FN	67.0
BERT-large	MetaST	60.1
BERT-large	PromptST	67.6
BERT-large	LiST	70.6
RoBERTa-base	Prompt FN	73.0
RoBERTa-base	MetaST	62.9
RoBERTa-base	PromptST	73.1
RoBERTa-base	LiST	76.4
RoBERTa-large	Prompt FN	77.6
RoBERTa-large	MetaST	62.6
RoBERTa-large	PromptST	77.2
RoBERTa-large	LiST	82.0
Table 13: Average performance over various backbones with with training labels  $|K| = 30$  (with unlabeled data). MetaST, PromptST and LiST are semi-supervised approaches.
Backbone	Approach	Average Acc
BERT-base	Prompt FN	64.4
BERT-base	MetaST	57.7
BERT-base	PromptST	64.9
BERT-base	LiST	66.5
BERT-large	Prompt FN	64.8
BERT-large	MetaST	57.7
BERT-large	PromptST	65.6
BERT-large	LiST	68.5
RoBERTa-base	Prompt FN	71.2
RoBERTa-base	MetaST	59.8
RoBERTa-base	PromptST	71.5
RoBERTa-base	LiST	75.1
RoBERTa-large	Prompt FN	75.4
RoBERTa-large	MetaST	58.9
RoBERTa-large	PromptST	74.8
RoBERTa-large	LiST	79.5
Table 14: Average performance over various backbones with training labels  $|K| = 20$  (with unlabeled data). MetaST, PromptST and LiST are semi-supervised approaches.
Backbone	Approach	Average Acc
BERT-base	Prompt FN	58.2
BERT-base	MetaST	52.4
BERT-base	PromptST	59.6
BERT-base	LiST	60.9
BERT-large	Prompt FN	59.4
BERT-large	MetaST	53.8
BERT-large	PromptST	59.6
BERT-large	LiST	62.1
RoBERTa-base	Prompt FN	66.8
RoBERTa-base	MetaST	54.1
RoBERTa-base	PromptST	66.5
RoBERTa-base	LiST	69.4
RoBERTa-large	Prompt FN	69.3
RoBERTa-large	MetaST	53.8
RoBERTa-large	PromptST	68.2
RoBERTa-large	LiST	72.8
Table 15: Average performance over various backbones with with training labels  $\left| K\right|  = {10}$  (with unlabeled data). MetaST, PromptST and LiST are semi-supervised approaches.
MNLI	RTE
LIST	73.5(2.8) / 75.0(3.7)	71.0(2.4)
w/o re-init	66.7(2.8) / 68.3(4.3)	69.0(4.9)
w/o re-weighting	72.9(3.4) / 74.2(4.5)	69.7(4.1)
w/o warmup	67.9(12.9) / 69.0(13.1)	69.2(4.5)
w/ hard pseudo-labels	71.7(3.8) / 73.0(5.4)	69.5(4.2)
w/o Adapter (Full Model)	73.6(2.7) / 74.8(2.7)	71.2(2.3)
Table 16: Ablation analysis of LiST with # of training data  $= {30}$  .
MNLI	RTE
LiST	71.8(2.3) / 73.0(3.1)	69.0(3.5)
w/o re-init	65.6(2.6) / 66.9(3.4)	66.5(3.7)
w/o re-weighting	70.7(4.1) / 71.8(4.6)	67.1 (5.6)
w/o warmup	66.9(5.4) / 68.3(5.7)	67.4(5.1)
w/ hard pseudo labels	69.9(3.6) / 71.4(3.7)	67.7(3.5)
w/o Adapter (Full Model)	66.6 (3.2) / 68.1 (3.6)	69.69 (5.29)
Table 17: Ablation analysis of LiST with # of training data = 20.
MNLI	RTE
LiST	65.0(4.5) / 66.3(4.9)	64.2(2.8)
w/o re-init	58.7(4.4) / 59.4(5.5)	58.8(4.0)
w/o re-weighting	63.8(5.8) / 64.5(6.6)	61.7(2.6)
w/o warmup	62.7(5.2) / 63.3(6.2)	61.7(4.8)
w/ hard pseudo labels	60.8(6.6) / 61.8 (6.8)	60.8(3.1)
w/o Adapter (Full model)	60.0 (3.7) / 61.1 (4.8)	62.4 (6.79)
Table 18: Ablation analysis of LiST with # of training data = 10.
Labels	Models	Avg	#Tunable
Params	MNLI (m/mm)
(acc)	RTE
(acc)	QQP
(acc)	SST-2
(acc)	Subj
(acc)	MPQA
(acc)
|K|=30	Classic FN	60.9	355M	38.0 (1.7) / 39.0 (3.1)	51.4 (3.7)	64.3 (8.1)	65.0 (11.5)	90.2 (2.2)	56.1 (5.3)
|K|=30 +Unlabeled Data	LIST w/ Classic FN	66.7	14M	39.9 (5.6) / 41.7 (7.6)	54.9 (1.4)	67.4 (7.0)	73.6 (9.9)	92.3 (1.1)	71.4 (4.7)
Table 19: Performance comparison of classic FN with RoBERTa-large as the encoder with standard deviation in parentheses. The best performance is shown in bold.
# of Training data	Approach	Average Acc (Six Tasks)
30	Full tuning	77.6
30	LiST Adapter	77.7
20	Full tuning	75.4
20	LiST Adapter	75.2
10	Full tuning	69.3
10	LiST Adapter	68.9
Table 20: Average Accuracy of Adapter w/ various number of training labels (No Semi-supervised Setting).