Pay Less Attention with Lightweight and Dynamic Convolutions

1 INTRODUCTION . There has been much recent progress in sequence modeling through recurrent neural networks ( RNN ; Sutskever et al . 2014 ; Bahdanau et al . 2015 ; Wu et al . 2016 ) , convolutional networks ( CNN ; Kalchbrenner et al . 2016 ; Gehring et al . 2016 ; 2017 ; Kaiser et al . 2017 ) and self-attention models ( Paulus et al. , 2017 ; Vaswani et al. , 2017 ) . RNNs integrate context information by updating a hidden state at every time-step , CNNs summarize a fixed size context through multiple layers , while as self-attention directly summarizes all context . Attention assigns context elements attention weights which define a weighted sum over context representations ( Bahdanau et al. , 2015 ; Sukhbaatar et al. , 2015 ; Chorowski et al. , 2015 ; Luong et al. , 2015 ) . Source-target attention summarizes information from another sequence such as in machine translation while as self-attention operates over the current sequence . Self-attention has been formulated as content-based where attention weights are computed by comparing the current time-step to all elements in the context ( Figure 1a ) . The ability to compute comparisons over such unrestricted context sizes are seen as a key characteristic of self-attention ( Vaswani et al. , 2017 ) . However , the ability of self-attention to model long-range dependencies has recently come into question ( Tang et al. , 2018 ) and the unlimited context size is computationally very challenging due to the quadratic complexity in the input length . Furthermore , in practice long sequences require the introduction of hierarchies ( Liu et al. , 2018 ) . ∗Work done during an internship at Facebook . 1Code and pre-trained models available at http : //github.com/pytorch/fairseq In this paper , we introduce lightweight convolutions which are depth-wise separable ( Sifre , 2014 ; Chollet , 2017 ; Kaiser et al. , 2017 ) , softmax-normalized and share weights over the channel dimension . The result is a convolution with several orders of magnitude fewer weights than a standard nonseparable convolution . Different to self-attention , lightweight convolutions reuse the same weights for context elements , regardless of the current time-step . Dynamic convolutions build on lightweight convolutions by predicting a different convolution kernel at every time-step . The kernel is a function of the current time-step only as opposed to the entire context as in self-attention ( Figure 1b ) . Dynamic convolutions are similar to locally connected layers in the sense that the weights change at every position , however , the difference is that weights are dynamically generated by the model rather than fixed after training ( LeCun et al. , 1998 ; Taigman et al. , 2014 ; Chen et al. , 2015 ) . Our approach also bears similarity to location-based attention which does not access the context to determine attention weights , however , we do not directly take the attention weights from the previous time-step into account ( Chorowski et al. , 2015 ; Luong et al. , 2015 ) . Shen et al . ( 2018b ) reduce complexity by performing attention within blocks of the input sequence and Shen et al . ( 2017 ; 2018c ) perform more fine-grained attention over each feature . Shen et al . ( 2018a ) and Gong et al . ( 2018 ) use input-dependent filters for text classification tasks . Our experiments show that lightweight convolutions perform competitively to strong self-attention results and that dynamic convolutions can perform even better . On WMT English-German translation dynamic convolutions achieve a new state of the art of 29.7 BLEU , on WMT English-French they match the best reported result in the literature , and on IWSLT German-English dynamic convolutions outperform self-attention by 0.8 BLEU . Dynamic convolutions achieve 20 % faster runtime than a highly-optimized self-attention baseline . For language modeling on the Billion word benchmark dynamic convolutions perform as well as or better than self-attention and on CNN-DailyMail abstractive document summarization we outperform a strong self-attention model . 2 BACKGROUND . We first outline sequence to sequence learning and self-attention . Our work builds on non-separable convolutions as well as depthwise separable convolutions . Sequence to sequence learning maps a source sequence to a target sequence via two separate networks such as in machine translation ( Sutskever et al. , 2014 ) . The encoder network computes representations for the source sequence such as an English sentence and the decoder network autoregressively generates a target sequence based on the encoder output . The self-attention module of Vaswani et al . ( 2017 ) applies three projections to the input X ∈ Rn×d to obtain key ( K ) , query ( Q ) , and value ( V ) representations , where n is the number of time steps , d the input/output dimension ( Figure 2a ) . It also defines a number of headsH where each head can learn separate attention weights over dk features and attend to different positions . The module computes dot-products between key/query pairs , scales to stabilize training , and then softmax normalizes the result . Finally , it computes a weighted sum using the output of the value projection ( V ) : Attention ( Q , K , V ) = softmax ( QKT√ dk ) V Depthwise convolutions perform a convolution independently over every channel . The number of parameters can be reduced from d2k to dk where k is the kernel width . The output O ∈ Rn×d of a depthwise convolution with weight W ∈ Rd×k for element i and output dimension c is defined as : Oi , c = DepthwiseConv ( X , Wc , : , i , c ) = k∑ j=1 Wc , j ·X ( i+j−d k+12 e ) , c 3 LIGHTWEIGHT CONVOLUTIONS . In this section , we introduce LightConv , a depthwise convolution which shares certain output channels and whose weights are normalized across the temporal dimension using a softmax . Compared to self-attention , LightConv has a fixed context window and it determines the importance of context elements with a set of weights that do not change over time steps . We will show that models equipped with lightweight convolutions show better generalization compared to regular convolutions and that they can be competitive to state-of-the-art self-attention models ( §6 ) . This is surprising because the common belief is that content-based self-attention mechanisms are necessary to obtaining stateof-the-art results in natural language processing applications . Furthermore , the low computational profile of LightConv enables us to formulate efficient dynamic convolutions ( §4 ) . LightConv computes the following for the i-th element in the sequence and output channel c : LightConv ( X , Wd cHd e , : , i , c ) = DepthwiseConv ( X , softmax ( Wd cHd e , : ) , i , c ) Weight sharing . We tie the parameters of every subsequent number of dH channels , which reduces the number of parameters by a factor of dH . As illustration , a regular convolution requires 7,340,032 ( d2 × k ) weights for d = 1024 and k = 7 , a depthwise separable convolution has 7,168 weights ( d × k ) , and with weight sharing , H = 16 , we have only 112 ( H × k ) weights . We will see that this vast reduction in the number of parameters is crucial to make dynamic convolutions possible on current hardware . Wang & Ji ( 2018 ) ties the weights of all channels ( H = 1 ) . Softmax-normalization . We normalize the weights W ∈ RH×k across the temporal dimension k using a softmax operation : softmax ( W ) h , j = expWh , j∑k j′=1 expWh , j′ Module . Figure 2b shows the architecture of the module where we integrate LightConv . We first apply an input projection mapping from dimension d to 2d , followed by a gated linear unit ( GLU ; Dauphin et al . 2017 ) , and the actual lightweight convolution . The GLU uses half of the inputs as gates by applying sigmoid units and then computes a pointwise product with the other inputs . We also apply an output projection of size WO ∈ Rd×d to the output of LightConv . Regularization . We found DropConnect to be a good regularizer for the LightConv module ( Wan et al. , 2013 ) . Specifically , we drop every entry of the normalized weights softmax ( W ) with probability p and divide it by 1− p during training . This amounts to removing some of the temporal information within a channel . Implementation . Existing CUDA primitives for convolutions did not perform very well to implement LightConv and we found the following solution faster on short sequences : We copy and expand the normalized weights W ∈ RH×k to a band matrix of size BH × n × n , where B is the batch size . We then reshape and transpose the inputs to size BH × n × dH , and perform a batch matrix multiplication to get the outputs . We expect a dedicated CUDA kernel to be much more efficient . 4 DYNAMIC CONVOLUTIONS . A dynamic convolution has kernels that vary over time as a learned function of the individual time steps . A dynamic version of standard convolutions would be impractical for current GPUs due to their large memory requirements . We address this problem by building on LightConv which drastically reduces the number of parameters ( §3 ) . DynamicConv takes the same form as LightConv but uses a time-step dependent kernel that is computed using a function f : Rd → RH×k : DynamicConv ( X , i , c ) = LightConv ( X , f ( Xi ) h , : , i , c ) we model f with a simple linear module with learned weights WQ ∈ RH×k×d , i.e. , f ( Xi ) =∑d c=1W Q h , j , cXi , c . Similar to self-attention , DynamicConv changes the weights assigned to context elements over time . However , the weights of DynamicConv do not depend on the entire context , they are a function of the current time-step only . Self-attention requires a quadratic number of operations in the sentence length to compute attention weights , while the computation of dynamic kernels for DynamicConv scales linearly in the sequence length . Our experiments ( §6 ) show that models using DynamicConv match or exceed the performance of state-of-the-art models that use context-based self-attention . This challenges the typical intuitions about the importance of content-based self-attention in natural language processing applications . 5 EXPERIMENTAL SETUP . 5.1 MODEL ARCHITECTURE . We use an encoder-decoder architecture for sequence to sequence learning ( Sutskever et al. , 2014 ) and we closely follow the architectural choices presented in Vaswani et al . ( 2017 ) . Our self-attention baseline is the fairseq re-implementation of the Transformer Big architecture ( Ott et al. , 2018 ) .2 The encoder and decoder networks have N blocks each . Encoder blocks contain two sub-blocks : The first is a self-attention module ( §2 ) , a LightConv module ( 3 ) , or a DynamicConv module ( §4 ) . The second sub-block is a feed-forward module : ReLU ( W 1X+b1 ) W 2+b2 whereW 1 ∈ Rd×dff , W 2 ∈ Rdff×d and d = 1024 , dff = 4096 unless otherwise stated . Sub-blocks are surrounded by residual connections ( He et al. , 2015 ) and layer normalization ( Ba et al. , 2016 ) . Decoder blocks are identical except that they have an additional source-target attention sub-block between the self-attention and feed-forward module . The source-target attention is equivalent to the self-attention module , except that the values and keys are projections over the encoder output for each source word . Words are fed to the encoder and decoder networks in d dimensional embeddings . We add sinusoidal position embeddings to encode the absolute position of each word in the sequence ( Kaiser et al. , 2017 ; Vaswani et al. , 2017 ) . The model computes a distribution over vocabulary V by transforming the decoder output via a linear layer with weights WV ∈ Rd×V followed by softmax normalization . LightConv and DynamicConv are identical to Transformer Big , except that self-attention modules are swapped with either fixed or dynamic convolutions . These models also use fewer parameters per block ( cf . Figure 2b and Figure 2c ) and we therefore increase the number of blocks toN = 7 for the encoder to roughly match the parameter count of Transformer Big . We generally set H = 16 . Both LightConv and DynamicConv set the the encoder and decoder kernel sizes to 3 , 7 , 15 , 31x4 for each block respectively ; except for the decoder where we have only three top layers with kernel size 31 .

Summary: The paper proposes a convolutional alternative to self-attention. To achieve this, the number of parameters of a typical convolution operation is first reduced by using a depth-wise approach (i.e. convolving only within each channel), and then further reduced by tying parameters across layers in a round-robin fashion. A softmax is applied to the filter weights, so that the operation computes weighted sums of its (local) input (LightConv).