Title: Pay Less Attention with Lightweight and Dynamic Convolutions
Authors: Felix Wu
Angela Fan
Alexei Baevski
Yann Dauphin
Michael Auli
Venue: ICLR 2019
Keywords: Deep learning
sequence to sequence learning
convolutional neural networks
generative models

Abstract: Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

TL;DR: Dynamic lightweight convolutions are competitive to self-attention on language tasks.

Review discussion:
The paper proposes a convolutional alternative to self-attention. To achieve this, the number of parameters of a typical convolution operation is first reduced by using a depth-wise approach (i.e. convolving only within each channel), and then further reduced by tying parameters across layers in a round-robin fashion. A softmax is applied to the filter weights, so that the operation computes weighted sums of its (local) input (LightConv).
Because the number of parameters is dramatically reduced now, they can be replaced by the output of an input-dependent linear layer (DynamicConv), which gives the resulting operation a "local attention" flavour. The weights depend only on the current position, as opposed to the attention weights in self-attention which depend on all positions. This implies that the operation is linear in the number of positions as opposed to quadratic, which is a significant advantage in terms of scaling and computation time.
In the paper, several NLP benchmarks (machine translation, language modeling) that were previously used to demonstrate the efficacy of self-attention models are tackled with models using LightConv and DynamicConv instead, and they are shown to be competitive across the board (with the number of model parameters kept approximately the same).
This paper is well-written and easy to follow. The proposed approach is explained and motivated well. The experiments are thorough and the results are convincing. I especially appreciated the ablation experiment for which results are shown in Table 3, which provides some useful insights beyond the main point of the paper. The fact that a linear time approach can match the performance of self-attention based models is a very promising and somewhat surprising result.
In section 5.3, I did not understand what "head band, next band, last band" refers to. I assume this is described in the anonymous paper that is cited, so I suppose this is an artifact of blind review.
Still, even with the reference unmasked it might be useful to add some context here.
The "head band, next band, last band" terminology is from https://openreview.net/forum?id=ByxZX20qFQ, which is presumably the cited anonymous paper.
Thank you for your comments. We improved the description of the adaptive softmax hyperparameters ('band' terminology) in the updated version of the paper. We hope this is clearer now.
We refer to different subsets of the vocabulary as 'bands'. The most frequent words are denoted as "head band", and so on.
Overall, this is a really good paper.
The authors propose an alternative to content based similarity for NL applications as compared to self-attention models by proposing the parameter and sequence length efficient Lightweight and Dynamic Convolutions.
The authors show, over various NL tasks like Translation, LM and Abstractive summarisation, the comparison of self attention models with Lightweight and Dynamic convolution layer.
The weight sharing was particularly interesting and can be seen as applying different heads for the same kernel.
The experimental results give strong evidence for these alternatives proposed by the authors.
The lightweight and dynamic convolution layers, both perform similar or better than the self-attention layer in all the tasks.
The WMT EnFr result is much better than all the other models, establishing a new state of the art.
Question for the authors:
1. Is the weight sharing within the kernel mostly for reducing computation?
If so, did you trying varying H size and measure how much that affects performance? What is surprising is that, in the ablation table the weight sharing increases the BLEU score by 0.1.
2. Did you run any experiments where the kernel size covers the whole sentence?
3. Since the number of parameters only change linearly wrt sequence length, did you try running this on datasets that have really long sequences to show the effectiveness of this approach further?
4. How important was softmax normalization for training?
Thank you for your fruitful comments.
Q1: For DynamicConv, weight sharing reduces both computation and memory footprint, while for LightConv, it only reduces memory footprint.  Yes, we did try using large H sizes;
however, the performance degrades and the memory footprint increases dramatically which prohibits us from using a large batch size. As a consequence, training becomes much slower.  For your information, DynamicConv with H=64 gets BLEU score 26.8 ± 0.1 on newstest2013 compared to 26.9 ± 0.2 with H=16 in Table 3.
Q2: We conducted an additional experiment based on your suggestion. We set the encoder kernel size to 237 and the decoder kernel size to 267 at each layer to cover the whole sequence. The BLEU score drops slightly to 26.7 ± 0.1. This is a small difference and we expect that slightly tuned hyperparameters would close the gap.
Q3: In section 6.4, we show experiments for document summarization (CNN/DailyMail) where the input sequence is capped at 400 words and the output sequence is 57 words on average with some examples having summaries of up to 478 words. Our results show that the model performs very well in this setting.
Q4: We found it very important as training diverged without softmax-normalization (see Note in Table 3) for DynamicConv. We added a comparison of softmax-normalization to various alternatives to Appendix A of the updated paper.
Furthermore, we are able to train the model without softmax-normalization with more aggressive gradient clipping, a lower learning rate (reducing it by a factor of 5) and more updates (increasing it by 5 times), but this slowed down training dramatically.
The authors present lightweight convolutions and dynamic convolutions, two significant advances over existing depthwise convolution sequence models, and demonstrate very strong results on machine translation, language modeling, and summarization. Their results go even further than those of the Transformer paper in countering the conventional wisdom that recurrence (or another way of directly modeling long-distance dependencies) is crucial for sequence-to-sequence tasks. Some things that I noticed:
- While you do cite "Depthwise Separable Convolutions for Neural Machine Translation" from Kaiser et al. (ICLR 2018), there are some missed opportunities to compare more directly to that paper (e.g., by comparing to their super-separable convolutions). Kaiser et al. somewhat slipped under the community's radar after the same group released the Transformer on arXiv a week later, but it is in some ways a more direct inspiration for your work than the Transformer paper itself.
- I'd like to see more analysis of the local self-attention ablation. It's fantastic to see such a well-executed ablation study, especially one that includes this important comparison, but I'd like to understand more about the advantages and drawbacks of local self-attention compared to dynamic convolutions. (For instance, dynamic convolutions are somewhat faster at inference time in your results, but I'm unsure if this is contingent on implementation choices or if it's inherent to the architecture.)
- From a systems and implementation perspective, it would be great to see some algorithm-level comparisons of parallelism and critical path length between dynamic convolutions and self-attention. My gut feeling is that dynamic convolutions significantly more amenable to parallelization on certain kinds of hardware, especially at train time, but that the caching that's possible in self-attention inference might make the approaches more comparable in terms of critical path latency at inference time; this doesn't necessarily line up with your results so far though.
- You mostly focus on inference time, but you're not always as clear about that as you could be; I'd also like to see train time numbers. Fairseq is incredibly fast on both sides (perhaps instead of just saying "highly optimized" you can point to a paper or blog post?)
- The nomenclature in this space makes me sad (not your fault). Other papers (particularly a series of three papers from Tao Shen at University of Technology Sydney and Tianyi Zhou at UW) have proposed architectures that are similarly intermediate between self-attention and (in their case 1x1) convolution, but have decided to call them variants of self-attention. I could easily imagine a world where one of these groups proposed exactly your approach but called it "Dynamic Local Self-Attention," or even a world where they've already done so but we can't find it among the zillions of self-attention variants proposed in the past year. Not sure if there's anything anyone can do about that, but perhaps it would be helpful to briefly cite/compare to some of the Shen/Zhou work.
- I think you should have tried a language modeling dataset with longer-term dependencies, like WikiText-103. Especially if the results were slightly weaker than Transformer, that would help place dynamic convolutions in the architecture trade-off space.
That last one is probably my most significant concern, and one that should be fairly easy to address.
But it's already a great paper.
Thank you for your fruitful comments.
Q: Comparison to "Depthwise Separable Convolutions for Neural Machine Translation" from Kaiser et al. (ICLR 2018).
Super-separable convolutions modify the pointwise operation (introducing groups) that follows depthwise convolutions, while we modify the latter (which focuses on aggregating the temporal information). We did try to introduce groups to the linear layers of the feed-forward block (that follows light/dynamic convolutions) but to reach the same accuracy, we had to increase the number of parameters of the model to a similar level as with the dense linear, at which point the network became slower.
Q: “I'd also like to see train time numbers”
Here are training times:
Self-attention, 17.5h (with or without limited window)
DynamicConv, 16.9h
LightConv, 16.5h
Our current implementation of dynamic convolutions is actually quite inefficient. We put the various convolution kernels in a sparse tensor that has only non-zero entries for the diagonal entry, thus using a lot of space. We expect a dedicated CUDA kernel to be more efficient. We are investigating such a kernel.
Note that batching is much more efficient during training which smooths out some of the speed advantages we see at test time. During inference batching is by far not as efficient due to repeated invocation of the decoder at every time step.
Q: “Other papers (particularly a series of three papers from Tao Shen at University of Technology Sydney and Tianyi Zhou at UW) have proposed architectures that are similarly intermediate between self-attention and (in their case 1x1) convolution”
We will discuss our work in the light of Shen & Zhou (2017, 2018) and also reference Ott et al. (2018) wrt fairseq speed.
Q: “You should have tried a language modeling dataset with longer-term dependencies”
In section 6.4, we show experiments for CNN/DailyMail document summarization which entails long input and output sequences. The input sequence is capped at 400 words and the output sequence is 57 words on average with some examples having summaries of up to 478 words.

Meta-review:
role: AC
data: metareview: Very solid work, recognized by all reviewers as worthy of acceptance. Additional readers also commented and there is interest in the open source implementation that the authors promise to provide.

Decision: Accept (Oral)