Abstract: Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a limited resource setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We test and verify the proposed attention method on three language modeling tasks (i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task (i.e., WMT-2016 English-German). Multi-linear attention can not only largely compress the model parameters but also obtain performance improvements, compared with a number of language modeling approaches, such as Transformer, Transformer-XL, and Transformer with tensor train decomposition.

Introduction: In NLP, Neural language model pre-training has shown to be effective for improving many tasks BIBREF0 , BIBREF1 . Transformer BIBREF2 is based solely on the attention mechanism, and dispensing with recurrent and convolutions entirely. At present, this model has received extensive attentions and plays an key role in many neural language models, such as BERT BIBREF0 , GPT BIBREF3 and Universal Transformer BIBREF4 . However, in Transformer based model, a lot of model parameters may cause problems in training and deploying these parameters in a limited resource setting. Thus, the compression of large neural pre-training language model has been an essential problem in NLP research. In literature, there are some compression methods BIBREF5 , BIBREF6 , BIBREF7 proposed. When the vocabulary is large, the corresponding weight matrices can be enormous. Tensorized embedding (TE) BIBREF5 uses the way of tensor-train BIBREF8 to compress the embedding layers in Transformer-XL BIBREF9 . In TE BIBREF5 , researchers only study the compression of input embedding layers, rathar than the attention layer. Recently, Block-Term Tensor Decomposition(BTD) BIBREF10 is used to compress recurrent neural networks (RNNs) BIBREF6 . Ye et al. BIBREF6 propose a compact flexible structure to deal with the large number of model parameters instead by high dimensional inputs in training recurrent neural networks (RNNs). This method greatly reduces the parameters of RNNs and improves their training efficiency. Still, the model only considers the input layer compression by the idea of low-rank approximation. On the other hand, some methods BIBREF7 , BIBREF11 aim to develop a specific structure on its weight matrices and are successful in compressing the pre-trained models. However, the new structure after compressing can not be integrated into the model. In Transformer, the multi-head attention is a key part and it is constructed by a large number of parameters. Specifically, Ashish et.al BIBREF2 compute the attention function on a set of queries simultaneously, packed together into a matrix $Q$ , while the keys and values are also packed together into matrices $K$ and $V$ , respectively. The attention function then adopts a no-linear function $softmax$ over three matrices $Q$ , $K$ and $V$ . There are two challenges to find a high-quality compression method to compress the multi-head attention in Transformer. First, the self-attention function in Transformer is a non-linear function, which makes it difficult to compress. In order to address this challenge, we first prove that the output of the attention function of the self-attention model BIBREF2 can be linearly represented by a group of orthonormal base vectors. $Q$ , $K$ and $V$ can be considered as factor matrices. Then, by initializing a low rank core tensor, we use Tucker-decomposition BIBREF12 , BIBREF13 to reconstruct a new attention representation. In order to construct the multi-head mechanism and compress the model, we use the method of Block-Term Tensor Decomposition (BTD), which is a combination of CP decomposition BIBREF14 and Tucker decomposition BIBREF12 . The difference is that three factor matrices $Q,~K$ and $V$ are shared in constructing each 3-order block tensor. This process can lead to reduce many parameters. The second challenge is that the attention model after compressing can not be directly integrated into the encoder and decoder framework of Transformer BIBREF2 , BIBREF9 . In order to address this challenge, there are three steps as follows. First, the average of each block tensor can be computed; Second, some matrices can be given by tensor split. Third, the concatenation of these matrices can serve as the input to the next layer network in Transformer. After that, it can be integrated into the encoder and decoder framework of Transformer BIBREF2 , BIBREF9 and trained end-to-end. Moreover, we also prove that the 3-order tensor can reconstruct the scaled dot-product attention in Transformer by a sum on a particular dimension. Our method combines two ideas which are the low-rank approximation and parameters sharing at the same time. Therefore, it achieves the higher compression ratios. Although the self-attention (i.e., scaled dot-product attention) in Transformer can be reconstructed, we do not consider reconstructing it and choose to split the 3-order tensor (the output of Multi-linear attention) which is helpful for improving the accuracy in experiments. Our major contributions of this paper are as follows: In order to validate the benefits of our model, we test it on two NLP tasks, namely language modeling and neural machine translation. In our experiments, the multi-head attention can be replaced by the proposed model, namely multi-linear attention. We have observed that the standard multi-head attention can be compressed with higher compression ratios on One-Billion dtaset. As a result, we show that multi-linear attention not only considerably reduces the number of parameters, but also achieve promising experiments results, especially in language modeling tasks.

Question: How big is performance improvement proposed methods are used?
Answer: Data augmentation (es)  improved Adv es by 20% comparing to baseline 
Data augmentation (cs) improved Adv cs by 16.5% comparing to baseline
Data augmentation (cs+es) improved both Adv cs and Adv es by at least 10% comparing to baseline 
All models show improvements over adversarial sets