SlowGuess/NAACL_2022_OCR huggingface dataset:
Larry Wasserman. 2006. *All of nonparametric statistics*. Springer Science & Business Media.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtopicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771.
Yun Yang, Anirban Bhattacharya, and Debdeep Pati. 2017. Frequentist coverage and sup-norm convergence rate in gaussian process regression. arXiv preprint arXiv:1708.04753.
- The WebNLG dataset consists of mapping sets of RDF triples to text. The training data are Data/Text pairs where the data is a set of (subject, property, object) triples. There are 9 categories extracted from DBpedia in the train and the development (dev) set, while the test set contains 5 more unseen categories, which can be used to evaluate the generalization of the adaptation methods. We adopt the official evaluation script and reports BLEU (Papineni et al., 2002), METEOR, (Lavie and Agarwal, 2007) and TER (Snover et al., 2006) $^{10}$ .
- CoQA is a large-scale conversational question answering dataset. It contains over 127K questions with answers collected from more than 8K conversations. The problem involves generating answers to the questions based on related conversation histories and documents. We follow the official evaluation script and use the macro-average F1 score of word overlap as the main evaluation metric (Reddy et al., 2019).
- CNN/DM is a benchmark for text summarization, involving more than 300K news articles provided by CNN and the Daily Mail. We report ROUGE-2 scores (Lin, 2004) as evaluation metrics.
- MNLI The Multi-Genre Natural Language Inference Corpus (Williams et al., 2018) provides sentence pairs of hypotheses and premises with entailment annotations. There are  $393k$  pairs in the training set,  $10k$  in the dev set, and another  $10k$  pairs in the test set. (Only the dev set is used in Table 2.) The premise sentences come from ten different sources, and the model performance can be evaluated on both the matched (in-domain) and mismatched (cross-domain) sections. In Table 2, we follow the setting used by Hu et al. (2021) and report mismatched accuracy as the metric.
- SST2 The Stanford Sentiment Treebank (Socher et al., 2013) is a corpus with fully labeled parse trees. In this corpus, 11,855 single sentences extracted from movie reviews were parsed with the Stanford parser (Klein and Manning, 2003), generating 215, 154 unique phrases from those parse
trees (3 human judges annotate each phrase). Wang et al. (2019) incorporated the task into the GLUE benchmark, with  $67k$  sentences in the training set,  $0.9k$  in the dev set, and  $1.8k$  instances in the dev set. (Only the dev set is used in Table 2.) The metric is the accuracy of the decision whether the sentiment of a review sentence is positive or negative.
B.1 General Training Settings
We avoid the sentence-level knowledge distillation trick used in (Lin et al., 2020), as it might interfere in analyzing our hypotheses. However, we retain the usage of "task embeddings" as they are required by the original GPT-2 model. These task embeddings act as specialized segment embeddings that indicate the different components of the text input (e.g., the three components of a triple in NLG, questions and answers in CoQA, etc.).<sup>11</sup>
We state the specific task embedding used in each task. For CoQA and CNN/DM, we follow the task embedding suggested by Lin et al. (2020); for WebNLG, we similarly set up the special tokens for the different components in the triples. We conclude the details of the special tokens in each dataset in Table B.4. Notably, the parameter budget for task embedding is negligible compared to the size of the aforementioned parameter-efficient adaptation techniques.
B.2 Hyper-parameters
We apply an AdamW optimizer and a linear learning rate scheduler with a 500-step warmup duration in training. At generation time, we use a greedy search for all the tasks, the same as Lin et al. (2020). For the choice of some hyperparameters, we mainly follow the setting used by Lin et al. (2020); Hu et al. (2021) and He et al. (2021), including the number of epochs and the argument for weight decay. Specifically, for WebNLG, CNN/DM, MNLI, and SST2, we train the model for 10 epochs; for CoQA, we train the model for 5 epochs. For the other important hyper-parameters, such as batch size and learning rate, we tune the hyper-parameters for different methods according to the loss on the validation set. For our Kernel-mix methods in Table 1, they share the same batch size and learning rate in each task. Specifically, for WebNLG, the learning
11The task embedding for the special tokens will also be updated during training, while we do not count them in Table 1.
10For TER, the lower the metric is, the better the performance is.
Datasets	Special tokens	# of trainable parameters for task embedding
WebNLG	<bos_webnlg>, <eos_webnlg>, <subject>, <property>, <object>, <target_webnlg>	6 * 768 = 4608
CoQA	<bos_qa>, <eos_qa>, <question>, <answer>, <document>	5 * 768 = 3840
CNN/DM	<bos_sm>, <eos_sm>, <source_sm>, <target_sm>	4 * 768 = 3072
Table B.4: The special tokens used in different tasks and the corresponding size of trainable parameters.
rate we use is 0.00125, and the batch size is 16; for CoQA, the learning rate we use is 0.005, and the batch size is 8; for CNN/DM, the learning rate we use is 0.001, and the batch size is 16; for MNLI, the learning rate we use is 0.0002, and the batch size is 32; for SST2, the learning rate we use is 0.0001, and the batch size is 16;
We train each variant for multiple independent runs to account for variability. In particular, for WebNLG, we train models over 5 runs, for CoQA 3 runs, for CNN/DM 2 runs, for MNLI 3 runs, and for SST2 3 runs.[12] The reported numbers in Tables 1 and 2 are the mean value averaged over the runs.
B.3 Implementation and Training Efficiency
All the models in this work are implemented by PyTorch. For the devices, we perform the distributed training using 8 Tesla V100 16GB GPUs. On WebNLG, it will take our method around 1 minute to finish one epoch; on CoQA, the time cost is around 20 minute / epoch; on CNN/DM, the training time per epoch will be 30 minutes. For the NLU tasks, the task implementation by He et al. (2021) cannot be adapted to distributed training (can only be trained with one graphic card), and thus the training time is longer: it will take our method around 2 hours to finish one epoch in MNLI, and 25 minutes in SST2.
We additionally report there is actually an implementation trick in Kernel-wise-lite. We can simply associate the  $h^{th}$  head's key matrix  $K^{(h)}$  to the computation of any weight matrix, say  $Q^{(h)}$ , as follows,
$$
\begin{array}{l} \boldsymbol {Q} ^ {(h)} = \boldsymbol {X} \boldsymbol {W} _ {q} ^ {(h)} + \boldsymbol {X} \boldsymbol {W} _ {k} ^ {(h)} \boldsymbol {B} _ {k} ^ {(h)} \boldsymbol {A} ^ {(h)} \\ = \boldsymbol {X} \boldsymbol {W} _ {q} ^ {(h)} + \boldsymbol {K} ^ {(h)} \boldsymbol {B} _ {k} ^ {(h)} \boldsymbol {A} ^ {(h)}. \\ \end{array}
$$
In that case we can reuse the given  $\pmb{K}$  to save the computation of the product  $X(W_k^{(h)}B_k^{(h)}A^{(h)})$ .
B.4 Specific settings for each method
We report the exact setting for the methods that need further explanation in this subsection. For Compacter, the bottleneck size of the adapter is 192, and the number of components is 4, as suggested by Karimi Mahabadi et al. (2021); for the original LoRA and the variants of our proposed methods, we summarize their settings in Table B.5. In this table, the numbers in columns Q_wise, V_wise, and O_wise are the rank used for Kernel-wise; if the number is followed by "(lite)", we apply Kernel-wise-lite with the listed rank to adjust the corresponding weight matrices. The numbers in columns Q_LoRA, V_LoRA, and O_LoRA are the rank of the update used as in the original LoRA. For Kernel-mix methods, the numbers in columns Q_wise and Q_LoRA (for example) will be nonzero.
C Partial Experimental Results Reported in LoRA (Hu et al., 2021)
For ease of reading, we copy Table 5 from the paper (Hu et al., 2021) as a piece of evidence to show "putting all the parameters in  $\Delta W_{q}$  or  $\Delta W_{k}$  results in significantly lower performance".
12We reduce the number of runs for larger datasets given computation budget.
Budget	Q_wise	Q_LoRA	V_wise	V_LoRA	O_wise	O_LoRA
LoRA-4	small	0	4	0	4	0	0
LoRA-54	intermediate	0	4	0	4	0	0
Kernel-mix-lite(qv)	tiny	1 (lite)	1	1 (lite)	1	0	0
Kernel-mix-lite(qv)	small	2 (lite)	2	1 (lite)	2	0	0
Kernel-mix(qvo)	intermediate	3	12	8 (lite)	8	8	8
Kernel-wise-lite(qv)	small	4 (lite)	0	4 (lite)	0	0	0
Kernel-wise(mq)	intermediate	12	0	4	0	0	0
Kernel-wise(mv)	intermediate	4	0	12	0	0	0
Kernel-wise(qvo)	intermediate	5	0	10 (lite)	0	10	0
Kernel-mix(qvo)	0.5% in Table 2	1	2	2 (lite)	4	2	4
Table B.5: The exact settings for the original LoRA and the variants of our proposed methods.
# of Trainable Parameters = 18M
Weight Type	Wq	Wk	Wv	Wo	Wq, Wk	Wq, Wv	Wq, Wk, Wv, Wo
Rank r	8	8	8	8	4	4	2
WikiSQL (±0.5%)	70.4	70.0	73.0	73.2	71.4	73.7	73.7
MultiNLI (±0.1%)	91.0	90.8	91.0	91.3	91.3	91.3	91.7
Table C.6: Validation accuracy provided by Hu et al. (2021, Table 5) on WikiSQL and MultiNLI.