Title: MUSE: PARALLEL MULTI-SCALE ATTENTION FOR SEQUENCE TO SEQUENCE LEARNING

Abstract: In sequence to sequence learning, the self-attention mechanism proves to be highly effective, and achieves significant improvements in many tasks. However, the self-attention mechanism is not without its own flaws. Although self-attention can model extremely long dependencies, the attention in deep layers tends to overconcentrate on a single token, leading to insufficient use of local information and difficultly in representing long sequences. In this work, we explore parallel multi-scale representation learning on sequence data, striving to capture both long-range and short-range language structures. To this end, we propose the Parallel MUlti-Scale attEntion (MUSE) and MUSE-simple. MUSE-simple contains the basic idea of parallel multi-scale representation learning, and it encodes the sequence in parallel, in terms of different scales with the help from self-attention, and pointwise transformation. MUSE builds on MUSE-simple and explores combining convolution and self-attention for learning sequence representations from more different scales. We focus on machine translation and the proposed approach achieves substantial performance improvements over Transformer, especially on long sequences. More importantly, we find that although conceptually simple, its success in practice requires intricate considerations, and the multi-scale attention must build on unified semantic space. Under common setting, the proposed model achieves substantial performance and outperforms all previous models on three main machine translation tasks. In addition, MUSE has potential for accelerating inference due to its parallelism.

1. INTRODUCTION:

In recent years, Transformer has been remarkably adept at sequence learning tasks like machine translation (Vaswani et al., 2017; Dehghani et al., 2018 ), text classification (Devlin et al., 2018; Yang et al., 2019) , language modeling (Sukhbaatar et al., 2019b; Dai et al., 2019) , etc. It is solely based on an attention mechanism that captures global dependencies between input tokens, dispensing with recurrence and convolutions entirely. The key idea of the self-attention mechanism is updating token representations based on a weighted sum of all input representations. However, recent research (Tang et al., 2018) has shown that the Transformer has surprising shortcomings in long sequence learning, exactly because of its use of self-attention. As shown in Figure 1 (a), in the task of machine translation, the performance of Transformer drops with the increase of the source sentence length, especially for long sequences. The reason is that the attention can be over-concentrated and disperse, as shown in Figure 1 (b), and only a small number of tokens are represented by attention. It may work fine for shorter sequences, but for longer sequences, it causes insufficient representation of information and brings difficulty for the model to comprehend the source information intactly. In recent work, local attention that constrains the attention to focus on only part of the sequences (Child et al., 2019; Sukhbaatar et al., 2019a) is used to address this problem. However, it costs self-attention the ability to capture long-range dependencies and also does not demonstrate effectiveness in sequence to sequence learning tasks.

To build a module with both inductive bias of local and global context modelling in sequence to sequence learning, we hybrid self-attention with convolution and present Parallel multi-scale attention called MUSE. It encodes inputs into hidden representations and then applies self-attention and depth-separable convolution transformations in parallel. The convolution compensates for the in- As we can see, the attention map is too dispersed to capture sufficient information. For example, "[EOS]", contributing little to word alignment, is surprisingly over attended. sufficient use of local information while the self-attention focuses on capturing the dependencies. Moreover, this parallel structure is highly extensible, i.e., new transformations can be easily introduced as new parallel branches, and is also favourable to parallel computation.

The main contributions are summarized as follows:

• We find that the attention mechanism alone suffers from dispersed weights and is not suitable for long sequence representation learning. The proposed method tries to address this problem and achieves much better performance on generating long sequence. • We propose a parallel multi-scale attention and explore a simple but efficient method to successfully combine convolution with self-attention all in one module.

• MUSE outperforms all previous models with same training data and the comparable model size, with state-of-the-art BLEU scores on three main machine translation tasks. • The proposed method enables parallel representation learning. Experiments show that the inference speed can be increased by 31% on GPUs.

2. MUSE: PARALLEL MULTI-SCALE ATTENTION:

Like other sequence-to-sequence models, MUSE also adopts an encoder-decoder framework. The encoder takes a sequence of word embeddings (x 1 , • • • , x n ) as input where n is the length of input. It transfers word embeddings to a sequence of hidden representation z

= (z 1 , • • • , z n ). Given z, the decoder is responsible for generating a sequence of text (y 1 , • • • , y m ) token by token.

The encoder is a stack of N MUSE modules. Residual mechanism and layer normalization are used to connect two adjacent layers. The decoder is similar to encoder, except that each MUSE module in the decoder not only captures features from the generated text representations but also performs attention over the output of the encoder stack through additional context attention. Residual mechanism and layer normalization are also used to connect two modules and two adjacent layers.

The key part in the proposed model is the MUSE module, which contains three main parts: selfattention for capturing global features, depth-wise separable convolution for capturing local features, and a position-wise feed-forward network for capturing token features. The module takes the output of (i -1) layer as input and generates the output representation in a fusion way:

X i = X i-1 + Attention(X i-1 ) + Conv(X i-1 ) + P ointwise(X i-1 )

(1) where "Attention" refers to self-attention, "Conv" refers to dynamic convolution, "Pointwise" refers to a position-wise feed-forward network. The followings list the details of each part. We also propose MUSE-simple, a simple version of MUSE, which generates the output representation similar to the MUST model except for that it dose not the include convolution operation:

EQUATION

2.1 ATTENTION MECHANISM FOR GLOBAL CONTEXT REPRESENTATION Self-attention is responsible for learning representations of global context. For a given input sequence X, it first projects X into three representations, key K, query Q, and value V . Then, it uses a self-attention mechanism to get the output representation:

EQUATION

Where W O , W Q , W K , and W V are projection parameters. The self-attention operation σ is the dot-production between key, query, and value pairs:

σ(Q 1 , K 1 , V 1 ) = softmax( Q 1 K T 1 √ d k )V 1 (4)

Note that we conduct a projecting operation over the value in our self-attention mechanism V 1 = V W V here.

2.2. CONVOLUTION FOR LOCAL CONTEXT MODELING:

We introduce convolution operations into MUSE to capture local context. To learn contextual sequence representations in the same hidden space, we choose depth-wise convolution (Chollet, 2017) (we denote it as DepthConv in the experiments) as the convolution operation because it includes two separate transformations, namely, point-wise projecting transformation and contextual transformation. It is because that original convolution operator is not separable, but DepthConv can share the same point-wise projecting transformation with self-attention mechanism. We choose dynamic convolution (Wu et al., 2019a) , the best variant of DepthConv, as our implementation.

Each convolution sub-module contains multiple cells with different kernel sizes. They are used for capturing different-range features. The output of the convolution cell with kernel size k is:

Conv k (X) = Depth conv k (V 2 )W out V 2 = XW V (5)

where W V and W out are parameters, W V is a point-wise projecting transformation matrix. The Depth conv refers to depth convolution in the work of Wu et al. (2019a) . For an input sequence X, the output O is computed as:

EQUATION

where d is the hidden size. Note that we conduct the same projecting operation over the input in our convolution mechanism V 2 = XW V here with that in self-attention mechanism.

Shared projection:

To learn contextual sequence representations in the same hidden space, the projection in the self-attention mechanism V 1 = V W V and that in the convolution mechanism V 2 = XW V is shared. because the shared projection can project the input feature into the same hidden space. If we conduct two independent projection here:

V 1 = V W V 1 and V 2 = XW V 2 , where W V

1 and W V 2 are two parameter matrices, we call it as separate projection. We will analyze the necessity of applying shared projection here instead of separate projection.

Dynamically Selected Convolution Kernels:

We introduce a gating mechanism to automatically select the weight of different convolution cells.

EQUATION

2.3. POINT-WISE FEED-FORWARD NETWORK FOR CAPTURING TOKEN REPRESENTATIONS:

To learn token level representations, MUSE concatenates an self-attention network with a positionwise feed-forward network at each layer. Since the linear transformations are the same across different positions, the position-wise feed-forward network can be seen as a token feature extractor.

P ointwise(X) = max(0, XW 1 + b 1 )W 2 + b 2 (8)

where W 1 , b 1 , W 2 , and b 2 are projection parameters.

3. EXPERIMENT:

We evaluate MUSE on four machine translation tasks. This section describes the datasets, experimental settings, detailed results, and analysis.

3.1. DATASETS:

WMT14 En-Fr and En-De datasets The WMT 2014 English-French translation dataset, consisting of 36M sentence pairs, is adopted as a big dataset to test our model. We use the standard split of development set and test set. We use newstest2014 as the test set and use newstest2012 +newstest2013 as the development set. Following Gehring et al. (2017) , we also adopt a joint source and target BPE factorization with the vocabulary size of 40K. For medium dataset, we borrow the setup of Vaswani et al. (2017) and adopt the WMT 2014 English-German translation dataset which consists of 4.5M sentence pairs, the BPE vocabulary size is set to 32K. The test and validation datasets we used are the same as Vaswani et al. (2017) .

IWSLT De-En and En-Vi datasets Besides, we perform experiments on two small IWSLT datasets to test the small version of MUSE with other comparable models. The IWSLT 2014 German-English translation dataset consists of 160k sentence pairs. We also adopt a joint source and target BPE factorization with the vocabulary size of 32K. The IWSLT 2015 English-Vietnamese translation dataset consists of 133K training sentence pairs. For the En-Vi task, we build a dictionary including all source and target tokens. The vocabulary size for English is 17.2K, and the vocabulary size for the Vietnamese is 6.8K.

3.2. EXPERIMENTAL SETTINGS:



3.2.1. MODEL:

For fair comparisons, we only compare models reported with the comparable model size and the same training data. We do not compare Wu et al. (2019b) because it is an ensemble method. We build MUSE-base and MUSE-large with the parameter size comparable to Transformer-base and Transformer-large. We adopt multi-head attention (Vaswani et al., 2017) as implementation of selfattention in MUSE module. The number of attention head is set to 4 for MUSE-base and 16 for MUSE-large. We also add the network architecture built by MUSE-simple in the similar way into the comparison.

MUSE consist of 12 residual blocks for encoder and 12 residual blocks for decoder, the dimension is set to 384 for MUSE-base and 768 for MUSE-large. The hidden dimension of non linear transformation is set to 768 for MUSE-base and 3072 for MUSE-large.

The MUSE-large is trained on 4 Titan RTX GPUs while the MUSE-base is trained on a single NVIDIA RTX 2080Ti GPU. The batch size is calculated at the token level, which is called dynamic batching (Vaswani et al., 2017) . We adopt dynamic convolution as the variant of depth-wise separable convolution. We tune the kernel size on the validation set. For convolution with a single kernel, we use the kernel size of 7 for all layers. In case of dynamic selected kernels, the kernel size is 3 for small kernels and 15 for large kernels for all layers.

3.2.2. TRAINING:

The training hyper-parameters are tuned on the validation set. 

MUSE-large:



MUSE-base:

We train and test MUSE-base on two small datasets, IWSLT 2014 De-En translation and IWSLT2015 En-Vi translation. Following Vaswani et al. (2017) , we use Adam optimizer with a learning rate of 0.001. We use the warmup mechanism and invert the learning rate decay with warmup updates of 4K. For the De-En dataset, we train the model for 20K steps with a batch size of 4K. The parameters are updated every 4 steps. The dropout rate is set to 0.4. For the En-Vi dataset, we train the model for 10K steps with a batch size of 4K. The parameters are also updated every 4 steps. The dropout rate is set to 0.3. We save checkpoints every epoch and average the last 10 checkpoints for inference.

3.2.3. EVALUATION:

During inference, we adopt beam search with a beam size of 5 for De-En, En-Fr and En-Vi translation tasks. The length penalty is set to 0.8 for En-Fr according to the validation results, 1 for the two small datasets following the default setting of Ott et al. (2019) . We do not tune beam width and length penalty but use the setting reported in Vaswani et al. (2017) . The BLEU1 metric is adopted to evaluate the model performance during evaluation.

3.3. RESULTS:

As shown in Table 1 , MUSE outperforms all previously models on En-De and En-Fr translation, including both state-of-the-art models of stand alone self-attention (Vaswani et al., 2017; Ott et al., 2018) , and convolutional models (Gehring et al., 2017; Kaiser et al., 2017; Wu et al., 2019a) . This result shows that either self-attention or convolution alone is not enough for sequence to sequence learning. The proposed parallel multi-scale attention improves over them both on En-De and En-Fr. It is shown in Table 1 and Table 2 that MUSE-simple which contains the basic idea of parallel multi-scale attention achieves state-of-the-art performance on three tasks.

3.4. HOW DO WE PROPOSE EFFECTIVE PARALLEL MULTI-SCALE ATTENTION?:

In this subsection we compare MUSE and its variants on IWSLT 2015 De-En translation to answer the question.

Does concatenating self-attention with convolution certainly improve the model? To bridge the gap between point-wise transformation which learns token level representations and self-attention which learns representations of global context, we introduce convolution to enhance our multi-scale attention. As we can see from the first experiment group of Table 3 , convolution is important in the parallel multi-scale attention. However, it is not easy to combine convolution and self-attention in one module to build better representations on sequence to sequence tasks. As shown in the first line of both second and third group of Furthermore, combining depth-wise separable convolution (in this work we choose its best variant dynamic convolution as implementation) is even worse than combining convolution.

Why do we choose DepthConv and what is the importance of sharing Projection of Depth-Conv and self-attention? We conjecture that convolution and self-attention both learn contextual sequence representations and they should share the point transformation and perform the contextual transformation in the same hidden space. We first project the input to a hidden representation and perform a variant of depth-wise convolution and self-attention transformations in parallel. The fist two experiments in third group of Table 3 show that validating the utility of sharing Projection in parallel multi-scale attention, shared projection gain 1.4 BLEU scores over separate projection, and bring improvement of 0.5 BLEU scores over MUSE-simple (without DepthConv).

How much is the kernel size? Comparative experiments show that the too large kernel harms performance both for DepthConv and convolution. Since there exists self-attention and point-wise transformations, simply applying the growing kernel size schedule proposed in SliceNet (Kaiser et al., 2017) doesn't work. Thus, we propose to use dynamically selected kernel size to let the learned network decide the kernel size for each layer.

3.5. FURTHER ANALYSIS:

Parallel multi-scale attention brings time efficiency on GPUs The underlying parallel structure (compared to the sequential structure in each block of Transformer) allows MUSE to be efficiently computed on GPUs. For example, we can combine small matrices into large matrices, and while it does not reduce the number of actual operations, it can be better paralleled by GPUs to speed up computation. Concretely, for each MUSE module, we first concentrate W Q , W K , W V of selfattention and W 1 of point feed-forward transformation into a single encoder matrix W Enc , and then perform transformation such as self-attention, depth-separable convolution, and nonlinear transformation, in parallel, to learn multi-scale representations in the hidden layer. W O , W 2 , W out can also be combined a single decoder matrix W Dec . The decoder of sequence to sequence architecture can be implemented similarly.

In Table 4 , we conduct comparisons to show the speed gains with the aforementioned implementation, the batch size is set to one sample per batch to simulate online inference environment. Under the settings, where the numbers of parameters are similar for MUSE and Transformer, about 31% increase in inference speed can be obtained. The experiments use MUSE with 6 MUSE-simple modules and Transformer with 6 base blocks. The hidden size is set to 512. It is worth noticing that for the MUSE structure used in the main experiments, ideally a similar speedup can be witnessed if the computing device is powerful enough. However, such is not the case in our preliminary experiments. We also need to point out the implementation is far from fully optimized and the results are only meant to demonstrate the feasibility of the procedure. Parallel multi-scale attention generates much better long sequence As demonstrated in Figure 3 , MUSE generates better sequences of various length than self-attention, but it is remarkably adept at generate long sequence, e.g. for sequence longer than 100, MUSE is two times better.

Lower layers prefer local context and higher layers prefer more contextual representations MUSE contains multiple dynamic convolution cells, whose streams are fused by a gated mechanism.

The weight for each dynamic cell is a scalar. Here we analyze the weight of different dynamic convolution cells in different layers. Figure 4 shows that as the layer depth increases, the weight of dynamic convolution cells with small kernel sizes gradually decreases. It demonstrates that lower layers prefer local features while higher layers prefer global features. It is corresponding to the finding in Ramachandran et al. (2019) . MUSE not only gains BLEU scores, but also generates more reasonable sentences and increases the translation quality. We conduct the case study on the De-En dataset and the cases are shown in Table 5 . In case 1, although the baseline transformer translates many correct words according to the source sentence, the translated sentence is not fluent at all. It indicates that Transformer does not capture the relationship between some words and their neighbors, such as "right" and "clap". By contrast, MUSE captures them well by combining local convolution with global selfattention. In case 2, the cause adverbial clause is correctly translated by MUSE while transformer misses the word "why" and fails to translate it.

4. RELATED WORK:

Sequence to sequence learning is an important task in machine learning. It evolves understanding and generating sequence. Machine translation is the touchstone of sequence to sequence learning. Traditional approaches usually adopt long-short term memory networks (Sutskever et al., 2014; Ma et al., 2018) to learn the representation of sequences. However, these models either are built upon auto-regressive structures requiring longer encoding time or perform worse on real-world natural language processing tasks. Recent studies explore convolutional neural networks (CNN) (Gehring et al., 2017) or self-attention (Vaswani et al., 2017) to support high-parallel sequence modeling and does not require auto-regressive structure during encoding, thus bringing large efficiency improvements. They are strong at capturing local or global dependencies.

There are several studies on combining self-attention and convolution. However, they do not surpass both convectional and self-attention mechanisms. Sukhbaatar et al. (2019b) propose to augment convolution with self attention by directly concentrating them in computer vision tasks. However, as demonstrated in Table 3 there method does not work for sequence to sequence learning task. Since state-of-the-art models on question answering tasks still consist on self-attention and do no adopt ideas in QAnet (Yu et al., 2018) . Both self-attention (Ott et al., 2018) and convolution (Wu et al., 2019a) outperforms Evolved transformer by near 2 BLEU scores on En-Fr translation. It seems that learning global and local context through stacking self-attention and convolution layers does not beat either self-attention or convolution models. In contrast, the proposed parallel multiscale attention outperforms previous convolution or self-attention based models on main translation tasks, showing its effectiveness for sequence to sequence learning.

5. CONCLUSION AND FUTURE WORK:

Although the self-attention mechanism has been prevalent in sequence modeling, we find that attention suffers from dispersed weights especially for long sequences, resulting from the insufficient local information.

To address this problem, we present Parallel Multi-scale Attention (MUSE) and MUSE-simple. MUSE-simple introduces the idea of parallel multi-scale attention into sequence to sequence learning. And MUSE fuses self-attention, convolution, and point-wise transformation together to explicitly learn global, local and token level sequence representations. Especially, we find from empirical results that the shared projection plays important part in its success, and is essential for our multiscale learning.

Beyond the inspiring new state-of-the-art results on three machine translation datasets, detailed analysis and model variants also verify the effectiveness of MUSE.

In future work, we would like to explore the detailed effects of shared projection on contextual representation learning. We are exited about future of parallel multi-scale attention and plan to apply this simple but effective idea to other tasks including image and speech.

:

https://github.com/moses-smt/mosesdecoder/blob/master/scripts/generic/multi-bleu.perl

Figures and Tables:

Figure 1: The left figure shows that the performance drops largely with the increase of sentence length on the De-En dataset. The right figure shows the attention map from the 3-th encoder layer.As we can see, the attention map is too dispersed to capture sufficient information. For example, "[EOS]", contributing little to word alignment, is surprisingly over attended.

Figure 2: Multi-scale attention hybrids point-wise transformation, convolution, and self-attention to learn multi-scale sequence representations in parallel. We project convolution and self-attention into the same space to learn contextual representations.

For training MUSE-large, followingOtt et al. (2018), parameters are updated every 32 steps. We train the model for 80K updates with a batch size of 5120 for En-Fr, and train the model for 30K updates with a batch size of 3584 for En-De. The dropout rate is set to 0.1 for En-Fr and 0.3 for En-De. We borrow the setup of optimizer fromWu et al. (2019a)  and use the cosine learning rate schedule with 10000 warmup steps. The max learning rate is set to 0.001 on En-De translation and 0.0007 on En-Fr translation. For checkpoint averaging, followingWu et al.  (2019a), we tune the average checkpoints for En-De translation tasks. For En-Fr translation, we do not average checkpoint but use the final single checkpoint.

Figure 3: BLEU scores of models on different groups with different source sentence lengths. The experiments are conducted on the De-En dataset. MUSE performs better than Transformer, especially on long sentences.

MUSE-large outperforms all previous models under the standard training and evaluation setting on WMT14 En-De and WMT14 En-Fr datasets.

Table content: <table><tr><td>Model</td><td colspan="2">En-Vi De-En</td></tr><tr><td>NBMT (Huang et al., 2017)</td><td>28.1</td><td>30.1</td></tr><tr><td>SACT (Lin et al., 2018)</td><td>29.1</td><td>-</td></tr><tr><td>NP2MT (Feng et al., 2018)</td><td>30.6</td><td>31.7</td></tr><tr><td>Fixup (Zhang et al., 2019)</td><td>-</td><td>34.5</td></tr><tr><td>DynamicConv (Wu et al., 2019a)</td><td>-</td><td>35.2</td></tr><tr><td>Macaron (Lu et al., 2019)</td><td>-</td><td>35.4</td></tr><tr><td>Local Joint Self-attention (Fonollosa et al., 2019)</td><td>-</td><td>35.7</td></tr><tr><td>MUSE-simple</td><td>30.7</td><td>35.8</td></tr><tr><td>MUSE</td><td>31.3</td><td>36.3</td></tr></table>



Table content: <table/>



Table content: <table><tr><td>Model</td></tr></table>

Comparisons between MUSE and its variants on the IWSLT 2015 De-En translation task.

Table content: <table/>

The comparison between the inference speed of MUSE and Transformer.

Table content: <table><tr><td colspan="2">Under review as a conference paper at ICLR 2020</td></tr><tr><td>Model</td><td>Inference Speed (tokens/s)</td></tr><tr><td>Transformer</td><td>132</td></tr><tr><td>MUSE</td><td>173</td></tr><tr><td>Acceleration</td><td>31%</td></tr></table>

Peer review (iclr2020):
The proposes a transformer layer that performs self-attention, feed-forward and dynamic-convolution all in parallel. The dynamic convolution consists of several different kernel sizes, which are then combined by learned weights. Experiments are done on three NMT tasks. In terms of novelty, there is very little. The model essentially combines DynamicConv from Wu et.al with self-attention. More importantly, it*s not clear why that*s better. The only direct comparison to Wu et.al. is the EN-FR translation task, but the proposed model doesn*t not outperform DynamicConv, even though having more parameters. Another novelty in the paper is placing feedforward in parallel to self-attention, which seems to improve the performance. This is probably why the model is more stable compared to Transformer because it has essentially 2x less depth, while maintaining the same capacity. As I read the paper, it felt bit rushed. For example, page 2 top says *language modeling*, but I can*t find any language modeling experiment. Figure 3 is never mentioned in the text. Also, the equation numbers in *weight tying* paragraph doesn*t match, and Eq8 is missing an operator (probably ReLU). In page6, the authors say *DynamicConv diverges*, but I can*t find that result. Also it said *The small version of MUSE still beats Transformer by a large margin.*, but where is that result? From table 3, it looks like they match in performance. Other comments: - The authors claim their architecture allow the feed-forward module better access to token features, but I think that*s only true for the first layer. Besides, Transformer has skip-connection that can directly feed token features to all layers. - Why W_in is tied to W^V? Why not W^K for example? - Why alpha is initialized to 1/n as if they are probabilities? When you pass alpha through softmax, any constant offset will have no effect on the output. - What are the kernel sizes used? - Type: *... dataset, We ...* - Figure 4 caption says *dynamically*, but my understanding is that the kernel weights are just learned (adaptive) and doesn*t dynamically change with inputs.