SlowGuess/NAACL_2022_OCR huggingface dataset:
$$
\left( \begin{array}{c} \boldsymbol {W} _ {o} ^ {(1)} \\ \vdots \\ \boldsymbol {W} _ {o} ^ {(N _ {h})} \end{array} \right),
$$
where, the  $i$ -th block  $\pmb{W}_{o}^{(h)} \in \mathbb{R}^{p \times N_{h}p}$  corresponds to head- $h$  (i.e.,  $\pmb{L}^{(h)}\pmb{V}^{(h)})$  in the attention output matrix, we can re-write the attention sub-layer as,
$$
\sum_ {h = 1} ^ {N _ {h}} \boldsymbol {L} ^ {(h)} \boldsymbol {V} ^ {(h)} \boldsymbol {W} _ {o} ^ {(h)}, \tag {6}
$$
and propose to take each summand as the complete form of a head (kernel estimator). We thus extend the coefficient part from the value matrices to the matrix products  $V^{(h)}W_{o}^{(h)},$  s, which naturally results in  $N$ -by-  $N_{h}p$  coefficients (with rank-  $p$ ).
4.3 Final Model
Combining the two pieces together, we report the concrete adaptation scheme under two settings:
- With very limited parameter budgets (less than  $0.2\%$  of the total PLM parameters), similar to LoRA, we modify  $W_{q}$  and  $W_{v}$  with equal parameter budgets using Kernel-mix-lite(qv). (The suffix (qv) means the method will adjust  $W_{q}$  and  $W_{v}$ .) In this case, we omit Guideline-2 and only incorporate Guideline-1 to apply head-specific updates.
- With intermediate parameter budgets (around  $1.6\%$  of the total parameters in the PLM), we suggest using Kernel-mix(qvo), which instead modifies  $\mathbf{W}_q, \mathbf{W}_v,$  and  $\mathbf{W}_o$ , assigning more budgets
to the coefficient part (Guideline-2). Given the increased parameter budgets, we allow Kernel-mix scheme for  $W_{q}$  and  $W_{o}$ , while continue to utilize Kernel-mix-lite scheme for  $W_{v}$ . Kernel-mix(qvo) incorporates both the guidelines.
While parameter-efficient tuning methods have been extensively studied for NLU tasks, their applicability towards NLG tasks is not well-known. This section performs empirical experiments of our proposed methods on three NLG tasks. To show the consistent effectiveness of our methods, we provide results on two NLU tasks as well.
We mainly evaluate the performance of our methods on NLG tasks, in which there is still a gap between fine-tuning and most parameter-efficient adaptation techniques (He et al., 2021). Specifically, we conduct our experiments on the following datasets: WebNLG-challenge (Gardent et al., 2017) for table-to-text tasks, CoQA (Reddy et al., 2019) for conversational question answering, and CNN/Daily-Mail (CNN/DM) (Hermann et al., 2015) for summarization (SUM). CNN/DM has only one domain, while CoQA has five domains<sup>9</sup>, and WebNLG has 14 domains. Descriptions of the datasets and evaluation metrics are provided in Appendix A.
Our experiments mainly follow the setting used by Lin et al. (2020), which takes GPT-2<sub>SMALL</sub> (124M parameters) (Wolf et al., 2019) as the backbone for all the NLG tasks. We choose the smaller model size as compared to the larger models used in other related studies it is generally difficult for smaller models to attain the same performance as full-model fine-tuning (Lester et al., 2021; Liu et al., 2021b). This creates a challenging testbed to evaluate our proposed approaches.
In addition to the NLG experiments, we also study two NLU tasks: MNLI (Williams et al., 2018) and SST2 (Socher et al., 2013), to show the performance of our methods on encoder-only transformers. The Multi-Genre Natural Language Inference (MNLI) Corpus (sentence pairs of hypotheses and premises with entailment annotations) will be given, and the task is to predict whether
The code release info will be available in this page.
<sup>9</sup>We use the official dev set as the test set and randomly select 500 examples from the train set as the new dev set.
Parameters to train	store	WebNLG	CoQA	SUM ROUGE-2
BLEU ↑	MET ↑	TER ↓	EM	F1
S	U	A	S	U	A	S	U	A
Full Budget = 100%
Fine-tuning	100.00%	100.00%	59.8	28.7	46.1	0.43	0.29	0.36	0.38	0.68	0.51	59.0	67.4	15.72
Tiny Budget < 0.1%
Adapter-4	0.08%	0.08%	51.7	35.6	44.4	0.38	0.32	0.35	0.44	0.58	0.51	49.9	58.8	14.18
Compacter	0.08%	0.08%	53.3	35.0	45.0	0.39	0.31	0.35	0.43	0.58	0.50	51.4	59.9	14.23
Bitfit	0.08%	0.08%	49.0	34.9	42.6	0.37	0.32	0.34	0.45	0.57	0.51	51.2	59.8	13.66
Kernel-mix-lite(qv)	0.07%	0.07%	51.0	36.7	44.5	0.38	0.32	0.35	0.43	0.54	0.48	53.1	61.6	14.27
Small Budget < 0.2%
Adapter-8	0.14%	0.14%	54.8	36.4	46.5	0.40	0.33	0.36	0.42	0.58	0.49	51.9	60.7	14.42
Prefix-tuning-8	7.92%	0.12%	49.2	35.6	43.0	0.37	0.31	0.34	0.45	0.56	0.50	50.1	58.6	14.18
LoRA-4	0.13%	0.13%	52.8	37.1	45.8	0.39	0.33	0.36	0.42	0.55	0.48	56.1	64.7	14.42
Kernel-mix-lite(qv)	0.13%	0.13%	53.8	37.2	46.3	0.39	0.33	0.36	0.41	0.54	0.48	55.4	63.9	14.52
Intermediate Budget < 2%
Adapter-108	1.62%	1.62%	59.5	34.1	48.2	0.42	0.32	0.38	0.38	0.61	0.49	57.7	66.4	15.22
Prefix-tuning-108	7.98%	1.60%	56.1	37.2	47.6	0.40	0.33	0.37	0.40	0.55	0.47	51.8	60.3	14.81
LoRA-54	1.61%	1.61%	54.8	36.9	46.7	0.40	0.33	0.37	0.41	0.55	0.47	57.2	65.7	15.29
Kernel-mix(qvo)	1.61%	1.61%	59.8*	36.7	49.3*	0.43*	0.33	0.38	0.37*	0.57	0.46*	59.9*	68.4*	15.34*
a We follow the notations in prefix-tuning (Li and Liang, 2021) that S, U, A represent SEEN, UNSEEN, and ALL respectively; SEEN categories are used in training; UNSEEN categories only appear in the test set; and ALL consists of all the categories.
Table 1: Performance  $(\%)$  on NLG tasks a. The methods are divided into 4 groups based on the number of parameters to store, and the methods in the same group have similar sizes. We boldface the best score in each group for different metrics. In the group of intermediate budget, we further conduct the significance tests between Kernel-wise and the best baseline for each metric (\* means the test p-value  $< 0.05$
the premise entails, contradicts, or is neutral to the hypothesis; the Stanford Sentiment Treebank (SST2) is composed of movie reviews and corresponding human-annotated sentiment and specifies a task to predict the sentiment of a review sentence (positive/negative). We implement the backbone under the setting used by He et al. (2021) and use RoBERTa<sub>BASE</sub> (125M parameters) (Liu et al., 2019) for both MNLI and SST2.
We compare our method with several other representative methods: fine-tuning (Howard and Ruder, 2018), adapters (Houlsby et al., 2019) used by Lin et al. (2020), Compacter (Mahabadi et al., 2021), Bitfit (Ben Zaken et al., 2021), prefix-tuning (Li and Liang, 2021), LoRA (Hu et al., 2021), and MAM-adapter (He et al., 2021). In Table 1 we use a postfix of adapters / LoRA / prefix-tuning to indicate their bottleneck size / rank of updates / prefix length, respectively. For instance, Adapter-4 means that the bottleneck size of the two-layer MLP in the inserted adapter is 4.
For some adaptation techniques, the number of parameters to train is not flexible to tune. For instance, Bitfit proposes to tune all the bias terms within the PLMs, and, as a result, the maximum parameters to tune are limited; as for Compacter, the weight matrices in the adapter modules are constructed through the Kronecker product, and the
parameter complexity is  $\mathcal{O}\left(\frac{L}{n} + n^3\right)$  (Mahabadi et al., 2021), where  $n$  is the size of a square matrix used in the Kronecker product and  $L$  is the number of layers. We choose a particular setting to make the number of trainable parameters in Compacter close to Bitfit and a tiny size adapter. The parameter size of Compacter is not further increased since a larger  $n$  would significantly retard the training.
For prefix-tuning, Li and Liang (2021) suggest to utilize a re-parametrization trick to mitigate its initialization issue, and therefore, the number of parameters to train will be much larger than the actual number of parameters to store, while these two numbers are the same for all other methods. In deciding the model size, we manage to make the number of parameters to store in prefix-tuning roughly the same as its adapter counterpart by adjusting the prefix length.
Table 1 compares our proposed methods against other baselines on the aforementioned generation tasks. The performance of our proposed Kernel-mix method on text classification is reported in Table 2. We summarize our observations as follows.
Tasks with long input. As shown in Table 1, all previous parameter-efficient methods fail to attain
comparable performance to fine-tuning on CoQA and CNN/DM tasks which have a longer input than the table-to-text generation task WebNLG. This indicates that current parameter-efficient methods still fall behind fine-tuning in those more challenging generation tasks with longer sequences. However, Kernel-mix, encloses this gap and even outperforms fine-tuning in some NLG tasks, e.g. the CoQA task, with solely  $1.61\%$  tunable parameters of GPT-2<sub>SMALL</sub>.
Impact of parameter sizes. The results in Table 1 show that overall, as the tunable parameter size increases, the performance of various parameter-efficient methods also increases, getting closer to fully fine-tuning. This indicates that a large enough parameter budget is still a prerequisite for the excellent performance of parameter-efficient adaptation methods. This finding can help us explain why the performance of Compacter can be better than Adapter-4 over all the tasks when the parameter budget is tiny (Tiny Budget  $< 0.1\%$  in the second group in Table 1), considering that the Compacter can construct a larger MLP than the adapter with the same parameter budget due to the usage of Kronecker product. Inspired by this finding, we can also clearly see the limitation of Bitfit and Compacter as their parameter budgets are constrained to be small and cannot be elevated of free will. As for the original LoRA, the impact of parameter sizes is somewhat tricky—LoRA-4 shows competitive performance while LoRA-54 is not improved as greatly as other methods on WebNLG and CoQA. A similar phenomenon on different datasets is also observed by Hu et al. (2021); He et al. (2021).
Performance of our proposed Kernel-mix(qvo). Our proposed Kernel-mix(qvo) can generally improve the performance on all three NLG datasets. On WebNLG, Kernel-mix(qvo) provides a 1.1 increase in BLEU score compared to Adapter-108 and a  $2.6\%$  increase compared to LoRA-54; on CoQA, our method is even more greatly better than LoRA-54, obtaining 2.7 exact-match and F1 improvement, and even outperforms fine-tuning by around  $1\%$  in both of the metrics; On CNN/DM, all the parameter-efficient methods have close performance, while through a test, we show our method Kernel-mix has a significantly higher Rouge-2 score than the best baseline, LoRA-54. Overall, Table 1 demonstrates that kernel-mix adaptation
better exploits the attention structure in PLMs and improves the overall generation quality under all three kinds of parameter budgets.
Method (# params)	MNLI	SST2
Fine-tuning (100%)	87.6±.4	94.6±.4
Bitfit (0.1%)	84.7	93.7
Prefix-tuning (0.5%)	86.3±.4	94.0±.1
LoRA (0.5%)	87.2±.4	94.2±.2
Adapter (0.5%)	87.2±.2	94.2±.1
MAM-Adapter (0.5%)	87.4±.3	94.2±.3
Kernel-mix(qvo) (0.5%)	87.4±.2	94.3±.3
Table 2: Accuracy on the dev set of MNLI and SST2. Bitfit numbers are copied from Ben Zaken et al. (2021), and all the other results (except for Kernel-mix) are from He et al. (2021, Table 2).
Performance on NLU tasks. Table 2 shows the performance of Kernel-mix(qvo) when it is extended to encoder-only transformers. For a fair comparison, we specify a new parameter budget (0.5%) for Kernel-mix(qvo), different from the previous settings in Table 1. With the new budget, Kernel-mix(qvo) attains close accuracy to the other parameter-efficient methods on both MNLI and SST2. We remark that the parameter budget used here is slightly tight for Kernel-mix(qvo), as the ranks assigned for head-wise adaptation (1 for  $W_{q}$  and 2 for  $W_{v}, W_{o}$ ) are limited (see Table B.5).
Besides the main results in Table 1, we also perform ablation studies to verify the effectiveness of our propositions. We additionally implement four variants of Kernel-wise to help ablate the effects of our proposed guidelines. Among the new variants, Kernel-wise-lite(qv), Kernel-wise(mq), and Kernel-wise(mv) only adjust  $W_{q}$ ,  $W_{v}$ ; Kernel-wise-lite(qv) takes the strategy in LoRA-4 to evenly assign parameters to  $W_{q}$  and  $W_{v}$ ; Kernel-wise(mq) leaves more budget to  $W_{q}$  than  $W_{v}$  with a ratio of 3:1, while Kernel-wise(mv) is set up in the reversed way. In contrast, Kernel-wise(qvo) simultaneously adjusts  $W_{q}$ ,  $W_{v}$ , and  $W_{o}$  (with a budget ratio of 5:1:10). The experimental results are summarized in Table 3. The settings of the variants designed for ablation are described in Appendix B.4.
Head-specific adaptation (Guideline-1). For "LoRA-4", the setting recommended by Hu et al. (2021), we compare it with its head-specific
Parameters to train	store	WebNLG	CoQA EM F1	SUM ROUGE-2
BLEU ↑ S U A S U A S U A
Budget < 0.2%
LoRA-4	0.13%	0.13%	52.8	37.1	45.8	0.39	0.33	0.36	0.42	0.55	0.48	56.1	64.7
Kernel-wise-lite(qv)	0.13%	0.13%	55.1	36.9	46.9	0.40	0.33	0.37	0.40	0.55	0.47	55.0	63.6
Budget < 2%
Kernel-wise(mq)	1.56%	1.56%	58.9	36.7	48.9	0.42	0.33	0.38	0.37	0.57	0.46	57.4	66.3
Kernel-wise(mv)	1.56%	1.56%	59.4	37.2	49.3	0.42	0.33	0.38	0.37	0.57	0.46	58.0	67.0
Kernel-wise(qvo)	1.61%	1.61%	59.5	36.1	49.0	0.43	0.33	0.38	0.37	0.58	0.47	59.1	68.0
Kernel-mix(qvo)	1.61%	1.61%	59.8	36.7	49.3	0.43	0.33	0.38	0.37	0.57	0.46	59.9	68.4
Table 3: Performance on new variants of Kernel-wise compared to LoRA-4 and our proposed Kernel-mix(qvo) (both copied from Table 1). The methods with similar budgets of tunable parameters are grouped. The exact settings of the methods to compare are illustrated in § 6.2 and Appendix B.4.
counterpart—Kernel-wise-lite(qv). In almost all the tasks, Kernel-wise-lite(qv) can attain better performance with the same number of parameters.
More parameters for the coefficient part (Guideline-2). Hu et al. (2021) (Section 7.1) indeed have already done some preliminary exploration to find the relatively more important weight matrices in transformers. Their experimental results (copied as Table C.6 in Appendix C) clearly show that "putting all the parameters in  $\Delta W_{q}$  or  $\Delta W_{k}$  results in significantly lower performance". In this work, we additionally show that by simply moving some trainable parameters from  $Q$ , the empirical kernel matrix part, to  $V$ , the coefficient part, Kernel-wise(mv) can improve the performance upon Kernel-wise(mq) as well.
Extending the scope of attention. To show the benefits of both adjusting  $W_{v}$  and  $W_{o}$ , we compare the new variant Kernel-wise(mv) against Kernel-wise(qvo). They both assign more budgets to the coefficient part; Kernel-wise(qvo) would update both  $W_{q}, W_{v}$  and  $W_{o}$ , while Kernel-wise(mv) only adjusts  $W_{q}, W_{v}$ . We can observe that Kernel-wise has better performance in most tasks.
Combining the shared and the head-specific basis. Lastly, we find that Kernel-mix(qvo) (our proposed method) outperforms Kernel-wise(qvo), which justifies combining the two types of basis, as opposed to pure head-specific adaptation.
In this work, we revisit the connection between the attention module and kernel estimators, and accordingly propose kernel-wise adaptation, which adopts the guidelines from kernel learning to strengthen the low-rank adaptation (LoRA). We verify that with the same parameter budgets, our proposed adaptation techniques can have better
performance on three generation tasks than the existing parameter-efficient methods, including adapters, prefix-tuning, and LoRA, and attain close accuracy on two classification tasks as well.
One possible extension of our work is combining our proposed method with other adapters in feed-forward sub-layers. In MAM-adapter, He et al. (2021) suggest applying prefix-tuning to adapt the parameters in self-attention sub-layers and assigning budgets to feed-forward sublayers as well; it can be beneficial to replace prefix-tuning with Kernel-mix for adaptation in the attention part.
Another direction of future research is the extension of our method to the feed-forward sub-layers, which are interpreted as key-value memories in recent work and behave like attention blocks (Geva et al., 2021). It will be interesting to study if kernel-specific guidelines could help design better adapters employed in the feed-forward layers.
We appreciate all the valuable feedback from the anonymous reviewers.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. arXiv e-prints, pages arXiv-2106.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. arXiv preprint arXiv:2005.14165.
Yifan Chen, Qi Zeng, Heng Ji, and Yun Yang. 2021. Skyformer: Remodel self-attention with gaussian kernel and nystr\`om method. Advances in Neural Information Processing Systems, 34.
Krzysztof Marcin Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Quincy Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. 2020. Rethinking attention with performers. In International Conference on Learning Representations.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Language Generation, pages 124-133.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. 2021. Towards a unified view of parameter-efficient transfer learning. arXiv preprint arXiv:2110.04366.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. Advances in neural information processing systems, 28:1693-1701.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799. PMLR.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv e-prints, pages arXiv-2106.
Dan Klein and Christopher D Manning. 2003. Accurate unlexicalized parsing. In Proceedings of the 41st annual meeting of the association for computational linguistics, pages 423-430.
Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the second workshop on statistical machine translation, pages 228-231.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), abs/2101.00190.
Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. 2020. Exploring versatile generative language model via parameter-efficient transfer learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 441-459.
Peiyu Liu, Ze-Feng Gao, Wayne Xin Zhao, Zhi-Yuan Xie, Zhong-Yi Lu, and Ji-Rong Wen. 2021a. Enabling lightweight fine-tuning for pre-trained language model compression based on matrix product operators. arXiv preprint arXiv:2106.02205.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021b. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. CoRR, abs/2110.07602.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021c. Gpt understands, too. arXiv:2103.10385.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. arXiv preprint arXiv:2106.04647.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Hao Peng, Nikolaos Pappas, Dani Yogatama, Roy Schwartz, Noah Smith, and Lingpeng Kong. 2020. Random feature attention. In International Conference on Learning Representations.
Jonas Pfeiffer, Ivan Vulic, Iryna Gurevych, and Sebastian Ruder. 2020. MAD-X: An Adapter-Based Framework for Multi-Task Cross-Linguual Transfer. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),
pages 7654-7673, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Andreas Rückle, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. arXiv preprint arXiv:2010.11918.
Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. 2001. A generalized representer theorem. In International conference on computational learning theory, pages 416-426. Springer.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Eliciting knowledge from language models using automatically generated prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223-231.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Yao-Hung Hubert Tsai, Shaojie Bai, Makoto Yamada, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Transformer dissection: An unified understanding for transformer's attention via the lens of kernel. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4344-4353.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In the Proceedings of ICLR.