SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2 shows the performance of mBERT models on IC and SL tasks with different types of late-fusion modules applied, along with fine-tune and last-layer baselines on four languages in the mA-TIS dataset. We observe a significant improvement in performance for both IC and SL tasks across all languages (an average of 6.4 absolute F1 score increase in IC and 3.6 absolute F1 score increase in SL) using attention-fusion compared to the last-layer model. Attention-fusion also achieves comparable performance to fine-tune model in both IC and SL tasks (an average of 0.12 absolute F1 score increase in IC and 0.13 absolute F1 score increase in SL). Linear-fusion and concat-fusion also outperforms the last-layer, demonstrating the effectiveness of utilizing intermediate layer representations of the encoder.
With the rising popularity of NLU systems, there is a need to expand them to new languages. An open challenge with language expansion is the scarcity of annotated data in the new language. A popular way to tackle this challenge is through transfer learning; thus, we examine the language
model. When the concatenated hidden representations are utilized by the decoder, it increases the size of FFN layers in the decoder, for a BERT-large model with 1024 hidden size and 24 layers, the first FFN layer in the decoder is  $(1024*24)*\text{decoder\_hidden\_units}$  which is  $24x$  the size of FFN layer in other approaches.
Figure 2: The distribution of attention weights for Intent Classification and Slot Labeling tasks on EN, ES, DE and FR languages of mATIS dataset.
transfer-ability of our proposed architecture in a few-shot learning setting compared to the baseline models. We measure F1 score on IC and SL tasks on four languages (EN, FR, DE, and ES) for the public mATIS dataset. We create a few-shot dataset for each one of the four languages by down-sampling the original training dataset to  $5\%$  of the original size. Table 3 summarizes the IC and SL performance of models fine-tuned on full-sized training data in the source language plus few-shot data in the target language, and evaluated on the target language. For example, EN -> FR indicates a model fine-tuned on full-size EN data and few-shot FR data, and evaluated on the FR dataset. We observe attention-fusion improves transfer-ability over last-layer baseline by a large margin (an average increase of 13 points F1 score for IC and 5.4 points F1 score for SL), achieving comparable performance with the fine-tune model on IC. Even though the F1 scores are lower than that of fine-tune model for SL in DE and ES, using attention-fusion allows for language expansion at a significantly lower cost, compared to the fine-tune baseline. The overall result suggests that attention-fusion can effectively improve knowledge transfer across languages.
5.3 Training Efficiency
In this section, we examine the training efficiency through number of trainable parameters and convergence speed of attention-fusion in comparison with other methods. A popular approach is Adapter in which the number of extra parameters is num_layers  $\times (2\times m\times d + m + d)$ , for BERT-large,  $m = 1024$ , num_layers  $= 24$ , with bottleneck dim of 256, results in 12.6 million trainable parameters. Bitfit adjusts the parameters of bias terms
Figure 3: The loss convergence speed of attention-fusion mechanism against other early and late fusion approaches on SST-2 task.
in Query, Key, Value matrices, as well as projection and feed-forward module among all layers, for BERT-large, this amounting to  $270\mathrm{K}$  trainable parameters. Prompt-tuning appends prompts to the first layer or to all layers (Liu et al., 2021), the preferred length of the prompt ranges from 20 to 100 tokens, for BERT-large, the trainable parameters is  $(1024\times$  prompt length) if prompt is inserted into the first layer, or  $(1024\times$  prompt length  $\times$  num layers) when inserted to all layers. In comparison, the attention-fusion mechanism is more efficient than other early-fusion based methods due to the following reasons:
- it uses a query vector of the same size as the pre-trained encoder hidden dimension (e.g., 1024 parameters for BERT-large) which adds significantly fewer parameters for training compared to early-fusion mechanisms
- the size of the fusion module does not grow with the number of layers in the pre-trained network unlike some other approaches
Table 4: Cross-lingual transfer: IC and SL results with pre-trained attention-fusion from EN on DE/ES/FR dataset. For baseline we train attention fusion with in-target language data. We report F1 scores on IC and SL tasks. Positive numbers indicate improvement over baseline.
Cross-lingual transfer	Intent Classification	Slot Labeling
DE	ES	FR	AVG	DE	ES	FR	AVG
in-target language fine-tuning (baseline)	95.96	96.68	97.18	96.52	95.58	91.06	94.79	94.48
in-target language fine-tuning with pre-trained fusion vector trained on EN	94.78	96.10	96.55	96.44	95.54	91.13	94.79	94.48
- it uses a late-fusion mechanism, in which the backward-pass and model update are performed only to the fusion module and decoder layers resulting in faster training.
In Figure 3, we report training loss over time for attention fusion and some other methods. We chose BitFit to represent an early-fusion approach due to its simplicity and effectiveness, and linear-fusion as an alternate late-fusion mechanism. We train all models with same number of GPUs and batch size, as well as adopt learning rate suggested by published paper. As shown in the plot, attention-fusion converges faster than BitFit, we hypothesis this is because late fusion does not need to back-propagate loss to all layers; hence making it faster to train and converge. Last-layer is the least performant given it doesn't harness intermediate representations; on the other hand, despite using intermediate layers, linear-fusion does not perform as good as attention fusion, suggesting the effectiveness and efficiency of the task-specific attention mechanism. $^{3}$
5.4 Analysis on Attention-fusion
In this section, we analyse the role and nature of the attention-fusion module. We visualize the distribution of attention weights after the softmax operation for different layers of mBERT in Figure 2. Along with the 12 layers of the mBERT-base encoder, we also attend to the embedding layer. Hence, the X-axis of all plots indicates layer 1-13, with 1 being the embedding layer and 13 being the 12th layer of BERT. The Y-axis denotes the attention weight associated with a layer. The attention weights across layers sum up to 1.0.
We show the attention weight distribution for IC and SL tasks in Figure 2 to investigate the learned attention patterns for different tasks. We observe
that the attention-fusion module attends to mBERT layers differently for different tasks. IC focuses on mid-late layers while SL focuses on early and mid-layers. The result demonstrates that the learned attention weights vary across tasks and thus attention-fusion can improve task adaptability with its flexibility in using intermediate representations. We hypothesize that IC relies on higher-level semantic information, while SL attends to both token-level embedding input from lower layers as well as contextual information from higher layers.
We also visualize the weights learned for IC and SL tasks on four languages Figure 2 to compare the attention patterns across languages. We observed that all languages learn similar attention weight distribution for different mBERT layers for each of the tasks. This observation explains the language transfer-ability of the attention-fusion model as described in the previous section. To further demonstrate that the attention-fusion module is task-specific and language-agnostic, we take a pre-trained attention-fusion module trained on EN language, freeze it and use it for different languages such as DE, FR and ES. We then compare this with the baseline in which the attention-fusion module is trained and evaluated on the same target language. As shown in Table 4, we observe comparable results against baseline; this suggests that the attention-fusion module is task-dependent and can transfer effectively across different languages.
With the rising popularity of transfer learning in NLU, the challenge of adapting pre-trained models to NLU tasks effectively and efficiently is becoming increasingly relevant. To address this challenge, we propose a light yet efficient task-specific attention-fusion module which enables parameter sharing and efficient fine-tuning for downstream tasks. We demonstrate that our proposed latefusion module achieves comparable performance
3For other approaches, our results showed attentionfusion is 4x faster than prompt-tuning v1& v2 and 1.5x faster than Adaptor when training with original released implementation. Due to differences in underlying training infrastructure, we did not report these results in Figure 3.
to other popular methods as well as the fine-tuning approach, while using less tune-able parameters per task. We also show that the task-specific attention-fusion module is transferable across languages, enabling language expansion work in NLU at a much lower cost.
Elad Ben-Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models.
Cristian Bucilua, Rich Caruana, and Alexandru Niculescu-Mizil. 2006. Model compression. In *In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining*, pages 535-541.
Jin Cao, Wang Jun, Shang-Wen Li, Kelly Vanee, and Wael Hamza. 2020. Style attuned pre-training and parameter efficient fine-tuning for spoken language understanding. In 2020 INTERSPEECH.
Qian Chen, Zhu Zhuo, and Wen Wang. 2019. Bert for joint intent classification and slot filling. arXiv preprint arXiv:1902.10909.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The pascal recognising textual entailment challenge. pages 177-190.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Prakhar Ganesh, Mohammad Ali Khan Yin Yang Deming Chen Marianne Winslett Hassan Sajjad Yao Chen, Xin Lou, and Preslav Nakov. 2020. Compressing large-scale transformer-based models: A case study on bert. In arXiv:2002.11985, 2020.
Mitchell A. Gordon, Kevin Duh, and Nicholas Andrews. 2020. Compressing bert: studying the effects of weight pruning on transfer learning. In CoRR, abs/2002.08307.
Demi Guo, Alexander M. Rush, and Yoon Kim. 2020. Parameter-efficient transfer learning with diff pruning.
Song Han, Huizi Mao, and William J. Dally. 2016. Compressing deep neural networks with pruning. In Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. In Proceedings of ICLR, 2016.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 2790-2799.
Chao-Wei Huang and Yun-Nung Chen. 2019. Adapting pretrained transformer to lattices for spoken language understanding. In 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. 2020. Tinybert: Distilling bert for natural language understanding. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4163-4174.
Olga Kovaleva, Alexey Romanov, Anna Rogers, and Anna Rumshisky. 2019. Revealing the dark secrets of bert. In Proceedings of the EMNLP 2019.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the EMNLP 2021.
Xiao Liu, Kaixuan Ji1, and Zhilin Yang Jie Tang Yicheng Fu1, Zhengxiao Du. 2021. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.
Yinhan Liu, Jingfei Du Mandar Joshi Danqi Chen Omer Levy Mike Lewis Luke Zettlemoyer Myle Ott, Naman Goyal, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. In CoRR, abs/1907.11692.
Saab Mansour and Haider Batool. 2020. Atis - seven languages. In LDC2021T04, Philadelphia: Linguistic Data Consortium, 2021.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227-2237.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8):9.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text.
Victor Sanh, Lysandre Debut, Julien Chaumont, and Thomas Wolf. 2019. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019a. Bert rediscovers the classical nlp pipeline. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4593-4601.
Ian Tenney, Alex Wang Adam Poliak R Thomas McCoy Najoung Kim Benjamin Van Durme Sam Bowman Dipanjan Das Patrick Xia, Berlin Chen, and Ellie Pavlick. 2019b. What do you learn from context? probing for sentence structure in contextualized word representations. In *In Proceedings of ICLR*, 2019.
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Gökhan Tür, Jerry Wright, Allen Gorin, Giuseppe Riccardi, and Dilek Hakkani-Tür. 2002. Improving spoken language understanding using word confusion networks. In Seventh International Conference on Spoken Language Processing.
Ashish Vaswani, Jakob Uszkoreit Llion Jones Aidan N. Gomez Lukasz Kaiser Noam Shazeer, Niki Parmar, and Illia Polosukhin. 2017. Attention is all you need. In CoRR, abs/1706.03762.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019a. Glue: A multi-task benchmark and analysis platform for natural language understanding.
Ziheng Wang, Jeremy Wohlwend, and Tao Lei. 2019b. Structured pruning of large language models. In arXiv:1910.04732, 2019b.
Zhilin Yang, Ruslan Salakhutdinov, and William W Cohen. 2017. Transfer learning for sequence tagging with hierarchical recurrent networks. arXiv preprint arXiv:1703.06345.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. 2019. Q8bert: Quantized 8bit bert. arXiv preprint arXiv:1910.06188.