SlowGuess/NAACL_2022_OCR huggingface dataset:
We provide a detailed result table for all compared methods in Section 4.5.3. Note that the M version model with  $m = 16$  and previous layer as input one is slightly higher than the results shown in Table 1(Cf. M-IDPG-PHM), this is because we tune the learning rate more carefully in Table 6 ( $lr \in \{1e^{-2}, 7e^{-3}, 5e^{-3}, 3e^{-3}, 1e^{-3}, 7e^{-4}, 5e^{-4}, 3e^{-4}, 1e^{-4}\}$ ) to seek the best performance each model can reach. While in Table 1, we tune the learning rate from  $\{5e^{-3}, 1e^{-3}, 5e^{-4}, 1e^{-4}\}$  to make the fair comparison with other models.
A.4 Supplementary Training for Single-layer IDPG
According to previous works (Phang et al., 2018; Wang et al., 2021), supplementing pre-trained LMs with rich data helps tasks with limited labels and stabilizes downstream fine-tuning. Following this idea, we also conduct intermediate training for single-layer IDPG.
Hyperparam	Supplimentary	Finetune	few-shot
#Layers	24	24	24
Hidden size	1024	1024	1024
FFN inner hidden size	4096	4096	4096
Attention heads	16	16	16
Attention head size	64	64	64
dropout	0.1	0.1	0.1
Learning Rate	linearly decayed	fixed	fixed
Peak Learning Rate	\( 1e^{-5} \)	\( \{5e^{-3}, 1e^{-3}, 5e^{-4}, 1e^{-4}\} \)	\( \{5e^{-4}, 1e^{-4}, 5e^{-5}, 1e^{-5}\} \)
Batch Size	32	{16, 32}	16
Weight Decay	0.1	0.1	0.1
Training Epoch	10	10 or 50	10 or 50
Adam \( \varepsilon \)	\( 1e^{-6} \)	\( 1e^{-6} \)	\( 1e^{-6} \)
Adam \( \beta_1 \)	0.9	0.9	0.9
Adam \( \beta_2 \)	0.98	0.98	0.98
Table 4: Hyperparameters for supplementary training, fine-tuning, few-shot fine-tuning.
Corpus	|Train|	|Validation|	Task	Evaluation Metrics
Single Sentence Tasks
CR	1,775	2,000	sentiment	accuracy
MR	8,662	2,000	sentiment	accuracy
SUBJ	8,000	2,000	sentiment	accuracy
MPQA	8,606	2,000	opinion polarity	accuracy
SST-2	67,349	1,821	sentiment analysis	accuracy
Sentence Pair Tasks
QNLI	104,743	5,463	NLI	accuracy
RTE	2,491	278	NLI	accuracy
MRPC	3,668	409	paraphrase	accuracy/F1
QQP	363,846	40,430	paraphrase	accuracy/F1
STS-B	5,749	1,500	sentence similarity	Pearson/Spearman corr.
Table 5: The datasets evaluated in this work.
Figure 5: Length difference of GLUE sentence pair datasets.
However, a drawback of supplementary training is that if the data distribution of the downstream tasks is quite different from the supplementary training task, i.e., MRPC vs. MNLI (Wang et al., 2019), it may harm the downstream performance. Figure 5 provides a comprehensive statistic among all sentence pair tasks in GLUE benchmark. For example, the length of the first sentence in MNLI is 9.8 longer than the second sentence on average, while this length difference in MRPC is only 0.6. One natural solution to smooth the length distribution difference between tasks is to insert prompt in both supplementary training and downstream finetuning stage. For example, assuming that we are adding a prompt with a length  $t = 5$  after the second sentence in the supplementary training stage on MNLI. Then, when fine-tuning downstream tasks
Table 6: Main results of different transfer learning method. Each methods are evaluated on full test sets (dev sets for GLUE tasks). We report average results across 5 runs with different initialization. We report the average of accuracy and F1 for both MRPC and QQP, and average of Pearson and Spearman correlation coefficients for STS-B. For all the other tasks, we report accuracy.
Method	m	MPQA	Subj	CR	MR	SST-2	QNLI	RTE	MRPC	STS-B	QQP	Avg
Input: Layer 0
S version	256	91.2±0.2	97.6±0.1	93.8±0.3	92.6±0.2	95.9±0.1	93.8±0.1	79.9±8.0	90.8±0.5	90.9±0.4	85.9±0.4	91.2
M version	256	91.2±0.3	97.5±0.1	93.6±0.3	92.7±0.2	95.7±0.2	94.3±0.1	85.5±1.0	91.8±0.3	91.4±0.2	87.0±0.4	92.1
L version	256	91.3±0.1	97.6±0.2	93.8±0.3	92.6±0.1	95.5±0.2	94.5±0.2	86.5±0.5	92.5±0.8	91.6±0.1	87.3±0.3	92.3
Input: Previous Layer
S version	256	91.2±0.2	97.5±0.1	93.5±0.3	92.6±0.1	95.8±0.3	94.0±0.1	83.4±1.5	91.9±0.3	91.1±0.3	86.9±0.2	91.8
M version	256	91.0±0.2	97.5±0.1	93.4±0.4	92.6±0.2	96.0±0.1	94.4±0.2	86.6±1.2	91.5±0.4	91.4±0.2	86.3±0.1	92.1
L version	256	91.3±0.2	97.4±0.0	93.3±0.3	92.5±0.2	95.8±0.1	94.5±0.3	86.9±0.8	92.1±0.4	91.7±0.2	87.1±0.2	92.3
Input: Previous Layer
S version	16	91.4±0.2	97.5±0.1	93.6±0.2	92.5±0.2	95.7±0.2	93.9±0.0	83.6±0.8	91.9±0.4	90.9±0.3	85.5±0.4	91.6
M version	16	91.2±0.2	97.5±0.1	93.4±0.3	92.6±0.3	96.0±0.3	94.5±0.1	83.5±0.7	92.3±0.2	91.4±0.4	87.1±0.1	92.0
such as MRPC, we concatenate the prompt after the first sentence. In this way, the length difference in MNLI and MRPC becomes more balanced: 4.8 vs.  $0.6 + 5 = 5.6$ . As shown in Figure 6, we test five different insertion positions (Pos 0–4) for sentence pair tasks and three different positions (Pos 0, 1, 4) for single sentence tasks. We further reduce the distribution difference by reconstructing the supplementary training data. We double the MNLI dataset by reordering the two sentences on one shard, and use the doubled dataset during intermediate training.
Figure 6: Insertion positions for sentence-pair tasks.
Architecture	Avg	Voting
PHM	86.1	86.9
+residual	85.9	86.7
+LayerNorm	86.1	87.1
+residual+LayerNorm	77.8	81.2
Table 7: Ablation study on generator architecture. We report average results and voting results across 5 runs.
A.5 Ablation study for single-layer IDPG
A.5.1 Generator Architecture Exploration
We explore three different architectures for the proposed PHM-based generator: (i) Residual: a residual structure (He et al., 2016) is applied to add the sentence representation to each generated tokens; (ii) LayerNorm: layer normalization (Ba
et al., 2016) is also added to normalize the generated token embedding; (iii) residual + layerNorm: a mixed model that uses both the residual component and LayerNorm. Note that, to balance the token embedding and sentence embedding, we apply LayerNorm to each embedding first, then after the add-up, use LayerNorm again to control the generated tokens. We observe that adding LayerNorm slightly improves the voting results, while residual performs slightly worse. One surprising result is that the mixed model of Residual and LayerNorm has significantly poorer performance.
A.5.2 Prompt Position
As we discussed in Section A.4, the prompt position has a direct impact on the prediction results. We conduct a comprehensive study of the prompt position for our proposed method in both supplementary training and downstream fine-tuning phases.
Looking at the prompt position in downstream tasks first, Figure 7(a) shows that for both standard prompt tuning and our proposed method, the best position is 0 for single-sentence tasks and 1 for sentence-pair tasks. This result is intuitive for single-sentence tasks since prompt in position 0 can be regarded as the premise and original input sentence as the hypothesis. For sentence-pair tasks, we hypothesize that inserting prompt into position 1 can better align the two input sentences. Figure 7(b) illustrates the effect of prompt position on the supplementary training phase. It is interesting that IDPG achieves best results in position 0 while the standard prompt-tuning achieves the best results in position 4 for both single-sentence and sentence-pair tasks.
Figure 7: Impact of prompt position on (a) downstream tasks; (b) supplementary training phase.
(a) The number of pairs of each group in Top-100 cosine similarity ranking.
Figure 8: The number of pairs of each group in Top-k cosine similarity ranking.
(b) The number of pairs of each group in Top-300 cosine similarity ranking.
A.6 Cosine Similarity Distributions in STS-B
We present the cosine similarity distributions when  $k = 100$  and  $k = 300$  in Figure 8a and in Figure 8b, respectively.
A.7 Ablation Study on Prompt Length
We present the impact of prompt length among several prompt tuning methods in Figure 9. IDPG shows its stability when scaling to larger models with longer prompts.
A.8 Potential Risks
Our proposed model IDPG is a novel efficient transfer learning method. It tunes small portion parameters while directly employs backbone model parameters without any changing. However, if the backbone model stored online is attacked, whether IDPG could still work well remains unknown. One should be careful to apply our proposed model and all other prompt tuning methods in high-stakes areas without a comprehensive test.
Figure 9: Impact of prompt length.