Sparse Transformer: Concentrated Attention Through Explicit Selection
Authors: Guangxiang Zhao
Junyang Lin
Zhiyuan Zhang
Xuancheng Ren
Xu Sun
Venue:  (2020)
Keywords: Attention
Transformer
Machine Translation
Natural Language Processing
Sparse
Sequence to sequence learning

Abstract: Self-attention-based Transformer has demonstrated the state-of-the-art performances in a number of natural language processing tasks. Self attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose a novel model called Sparse Transformer. Sparse Transformer is able to improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on a series of natural language processing tasks, including neural machine translation, image captioning, and language modeling, all demonstrate the advantages of Sparse Transformer in model performance. 
  Sparse Transformer reaches the state-of-the-art performances in the IWSLT 2015 English-to-Vietnamese translation and IWSLT 2014 German-to-English translation. In addition, we conduct qualitative analysis to account for Sparse Transformer's superior performance. 

Under review as a conference paper at ICLR 2020
EXPLICIT SPARSE TRANSFORMER: CONCENTRATED ATTENTION THROUGH EXPLICIT SELECTION
Anonymous authors Paper under double-blind review
ABSTRACT
seffatteention-base Transformer has demonstrated the state-of-the-art perfor- mances in number of natural language processing tasks. Self-attention is able to model long-term dependencies, but it may suffer from the extraction of irrelevant information in the context. To tackle the problem, we propose novel model called Explicit Sparse Transformer. Explicit Sparse Transformer able improve the concentration of attention on the global context through an explicit selection of the most relevant segments. Extensive experimental results on series of natural language processing and computer vision tasks, including neural machine trans- lation, image captioning, and language modeling, all demonstrate the advantages of Explicit Sparse Transformer in model performance. We also show that our proposed sparse attention method achieves comparable better results than the previous sparse attention method, but significantly reduces training and testing time. For example, the inference speed is twice that of sparsemax Transformer model.
INTRODUCTION
Understanding natural language requires the ability to pay attention to the most relevant information. For example, people tend focus on the most relevant segments to search for the answers their questions in mind during reading. However, retrieving problems may occur if irrelevant segments impose negative impacts on reading comprehension. Such distraction hinders the understanding process, which calls for an effective attention.
This principle is also applicable to the computation systems for natural language. Attention has been vital component of the models for natural language understanding and natural language generation. Recently, Vaswani et al. (2017) proposed Transformer, model based on the attention mechanism Transformer has shown outstanding performance in natural language generation tasks. More recently, the success of BERT (Devlin et al.] 2018) in natural language processing shows the great usefulness of both the attention mechanism and the framework of Transformer.
for Neural Machine
However, the attention in vanilla Transformer has obvious drawback, as the Transformer assigns credits to all components of the context. This causes a lack of focus. As illustrated in Figure [ the attention in vanilla Transformer assigns high credits many irrelevant words, while in Explicit Sparse Transformer, it concentrates on the most relevant k words. For the word "tim", the most related words should be "heart" and the immediate words. Yet the attention in vanilla Transformer does not focus them but gives credits to some irrelevant words such as "him".
Recent works have studied applying sparse attention in Transformer model. However, they either add local attention constraints (Child et al. 2019) which break long term dependency or hurt the time efficiency (Martins Astudillo 20166. Inspired by Ke et al. (2018) which introduce sparse credit assignment to the LSTM model, we propose novel model called Explicit Sparse Transformer which is equipped with our sparse attention mechanism. We implement an explicit selection method based on top-k selection. Unlike vanilla Transforner, Explicit Sparse Transformer only pays attention to the most contributive states. Thus Explicit Sparse Transformer can perform more concentrated attention than vanilla Transformer.

---

Under review as a conference paper at ICLR 2020
0.6
Shanted
Nem
heart
and
alled
Nm
wly
dre
youl
relppng
mre
Lquatt
(target word, means 'heart' in vietnamese)
Figure 1: Illustration of self-attention in the models. The orange bar denotes the attention score of our proposed model while the blue denotes the attention scores of the vanilla Transformer. The orange line denotes the attention between the target word "tim" and the selected top-k positions in the sequence. In the attention of vanilla Transformer, "tim" assigns too many non-zero attention scores to the irrelevant words. But for the proposal, the top-k largest attention scores removes the distraction from irrelevant words and the attention becomes concentrated.
We first validate our methods on three tasks. For further investigation, we compare our methods with previous sparse attention methods and experimentally answer how to choose in series of qualitative analyses. We are surprised to find that the proposed sparse attention method can also help with training as regularization method. Visual analysis shows that Explicit Sparse Transformer exhibits higher potential in performing high-quality alignment. The contributions of this paper presented below:
We propose a novel model called Explicit Sparse Transformer, which enhances the concen- tration of the Transformer's attention through explicit selection.
We conducted extensive experiments on three natural language processing tasks, including Neural Machine Translation, Image Captioning and Language Modeling. Compared with vanilla Transformer, Explicit Sparse Transformer demonstrates better performances in the above three tasks. Specifically, our model reaches the state-of-the-arr performances in the IWSLT 2015 English-to- Viii-toooiienneeeeee translation.
Compared to previous sparse attention methods for transformers, methods much faster in training and testing, and achieves better results.
PREMIERS
The review to attention mechanism and the attention- framework Transformer can be found in Appendix A.1
3 EXPLICIT SPARSE TRANSFORMER
Lack of concentration in the attention can lead to the failure of relevant information extraction. To this end, we propose a novel model. Explicit Sparse Transformer, which enables the focus on only few elements through explicit selection. Compared with the conventional attention, credit will be assigned to the value that is not highly correlated to the query. We provide a comparison between the attention of vanilla Transformer and that of Explicit Sparse Transformer in Figure
2

---

Under review as a conference paper at ICLR 2020
Output
Te--A welcttonn
Output
Softrma
Fri
Softmax
Sparsification
AgO
Softrc
Score
Score
Origial
Figure 2: The comparison between the attentions of vanilla Transformer and Explicit Sparse Trans- former and the illustration of the attention module Explicit Sparse Transformer. With the mask based on top-k selection and softmax function, only the most contributive elements are assigned with probabilities.
Explicit Sparse Transformer still based on the Transformer framework. The difference in the implementation of self-attention. The attention is degenerated to the sparse attention through top-k selection. In this way. the most contributive components for attention are reserved and the other irrelevant information are removed. This selective method is effective in preserving important information and removing noise. The attention can be much more concentrated on the most contributive elements of value. In the following, first introduce the sparsification in self-attention and then extend to context attention.
In the unihead alffaattetion, the key components, the query Qllo. d], key K[k,, d] and value Vily, d], are the linear transformation of the source context, namely the input of each layer, where Q = Waæ, K Wx and V = Wyx. Explicit Sparse Transformer first generates the attention scores P as demonstrated below:
p= QKT
Vd
Then the model evaluates the values of the scores / based on the hypothesis that scores with larger values demonstrate higher relevance. The sparse attention masking operation .M(-) is implemented upon P in order select the top-k contributive elements. Specifically, we select the largest element of each row in P and record their positions in the position matrix (i,j), where & is hyperparameter. To be specific, say k-th largest value row i 4,, the value of the j-th component is larger than t,, the position (i.j) is recorded. We concatenate the threshold value of cach to form vector t [t,, t2. . ttol. The masking functions M(..)) is illustrated as follows:
M(P,k) = { Pij if P.j 2 t. (k-th largest value of row i)
if P, 4 (k-th largest value of row i)
With the top-k selection, the high attention scores are selected through an explicit way. This is different from dropout which randomly abandons the scores. Such explicit selection not only guarantee the preservation of important components, but also simplify the model since is usually a small number such as 8, detailed analysis can be found in 5.2 The next step after top-k selection is normalization:
= softmax(MM(,kb)
where A refers to the normalized scores. As the scores that are smaller than the top k largest scores assigned with negative infinity by the masking function M(.-), -) their normalized scores, namely the probabilities, approximate 0. We show the back-propagation process of Top-k selection A.3 The output representation of self-attention C can be computed below:
C AV
3

---

Under review as a conference paper at ICLR 2020
Model
En-De
En-Vi De-En
ConvS2S Geehinn al.
2017
25.2
Actor-Critic Bahdanau
al. 2017
28.5
NPMT+LM Huang ct al.
28.1
30.1
SACT (Lin ct a.. 2018)
29.1
Var-Attn (Deng al.] 2018
33.7
NP2MT Feng
(2018)
30.6
31.7
Transformer (Vaswant
2017
RNMT (Chen ct al.
2018)
28.5
Fixup Zhang et al.
2019)
29.3
34.5
Weighted Transtormer (Ahmed et al.
2017
28.9
Universal Transformer [Dehghant
2018
Layer-wisc Coordination (He 2018)
Transformer(rclative position) (Shaw ct a
2018
29.2
Transformer (Ott et al. 2018
29.3
Dy DynamicCony (wu el al. 2019)
29.7
35.2
Local Joint Self-attention (Fonollosa el al.
2019
29.7
35.7
Trapl))merimpll.)
29.1
30.6
35.3
Explicit Sparse Transformer
29.4
35.6
Table 1: Results on the En-De, En- Vi and De-En test sets. Compared with the baseline models, Explicit Sparse Transformer reaches improved performances, and it achieves the state-of-the-art performances in En- Vi and De-En.
The output is the expectation of the value following the sparsified distribution A. Following the distribution the selected components, the attention in the Explicit Sparse Transformer model can obtain more focused attention. Also, such sparse attention but different from the self-attention mechanism, the Q is no longer the linear transformation of the source context but the decoding states In the implementation, we replace Q with Wøs, where Wq is still learnable matrix.
extend to context attention. Resembling
In brief, the attention in our proposed Explicit Sparse Transformer sparsifies the attention weights. The attention can then become focused on the most contributive elements, and it is compatible both self-attention and context attention. The simple implementation of this method in the Appendix A.4
RESULTS
We conducted series experiments on three natural language processing tasks, including neural machine translation, image captioning and language modeling. Detailed experimental settings are in Appendix
A.2 A.2
4.1 NEURAL MACHINE TRANSLATION
Dataset To evaluate the performance of Explicit Sparse Transformer in NMT, we conducted experiments on three NMT tasks, English-to-German translation (En-De) with a large dataset, English-to- Vietnamese (En-Vi Vi) translation and German-to-Engee datasets of medium size. For En-De, we trained Explicit Sparse Transformer on the standard dataset for WMT 2014 En-De translation. The dataset consists of around 4.5 million sentence pairs. The source and target languages share vocabulary of 32K sub-word units. We used the newstest 2013 for validation and the newstest 2014 as our test set. We report the results on the test set.
translation (De-En) with two
For En-Vi, we trained our model on the dataset in IWSLT 2015 (Cettolo et al. 2014). The dataset consists of around 133K sentence pairs from translated TED talks. The vocabulary size for source language is around 17,200 and that for target language is around 7,800. We used tst20/2 for validation. and tst2013 for testing and report the testing results. For De-En, we used the dataset in IWSLT 2014. The training set contains 160K sentence pairs and the validation set contains 7K sentences.
4

---

Under review as a conference paper at ICLR 2020
Model
BLEU-4
METEOR
CIDEr
SAT
Bazzani et al.
2018b
28.2
24.8
92.3
SCST
Rennie
2017
32.8
26.7
106.5
NBT
Lu al. 2018
34.7
27.1
107.2
AdaAt
Lu ct a
33.2
26,6
108.5
ARNN
Bazzami et al.
2018a
33.9
27.6
109.8
Transformer
35.3
27.7
113.1
UpDown Anderson (2018)
36.2
27.0
113.5
Explicit Sparse Transformer
35.7
28.0
113.8
Table 2:
Results on the MSCOCO Karpathy test split.
Following Edunov al. (2018), we used the same test set with around 7K sentences. The data were preprocessed with byte-pale -air encoding (Sennrich et al. 2016). The vocabulary size is 14,000.
Result Table presents the results of the baselines and our Explicit Sparse Transformer on three datasets. For En-De, Transformer-basee models outperform the previous methods. Compared with the result of Transformer (Vaswani et al., 2017). Explicit Sparse Transformer reaches 29.4 in BLEU score evaluation, outperforming vanilla Transformer by 0.3 BLEU score. For En Vi. vanilla Transformed reaches 30.2, outperforming the taae--of-the-art method (Huang et al. 2017). Our model, Explicit Sparse Transformer, achieves new state-of-the-art performance, 31.1, by margin of 0.5 over vanilla Transformer. For De-En, we demonstrate that Transformer-based models outperform the other baselines. Compared with Transformer, our Explicit Sparse Transformer reaches better performance, 35.6. Its advantage is +0.3. To the best of our knowledge, Explicit Sparse Transformer reaches top line performance on the dataset.
4.2 IMAGE CAPTIONING
Dataset We evaluated our approach on the image captioning task. Image captioning is a task that combines image understanding and language generation. We conducted experiments on the Microsoft COCO 2014 dataset (Chen et al. 2015a). It contains 123,287 images, each of which paired with descriptive sentences. We report the results and evaluate the image captioning model on the MSCOCO 2014 test set for image captioning. We used the publicly-available splits provided by Karpathy & Li 2015). The validation set and test set both contain 5,000 images.
Table 2 shows the results of the baseline models and Explicit Sparse Transformer on the COCO Karpathy test split. Transformer outperforms the mentioned baseline models, Explicit Sparse Transformer outperforms the implemented Transformer by +0.4 in terms of BLEU-4, +0.3 in terms of METEOR, +0.7 terms of CIDEr. which consistently proves effectiveness Image Captioning.
Result
4.3 LANGUAGE MODELING
Dataset E is large-scale dataset for character-level language modeling. It contains 100M bytes of unprocessed Wikipedia texts. The inputs include Latin alphabets, non-Latin alphabets, XML markups and special characters. The vocabulary size 205 tokens, including one for unknown characters. We used the same preprocessing method following Chung et al. (2015). The training contains 90M bytes of data, and the validation set and the test set contains 5M respectively.
Result Table 3 shows the results of the baseline models and Explicit Sparse Transformer-Xi XI. on the test set of enwiki8. Compared with the other strong baselines, Transformer-Xi I can reach better performance, and Explicit Sparse Transformer outperforms Transformer-XX XL with an advantage.
'While we did not the results of Transformer on En- Vi, we reimplemented our vanilla Transformer with the same setting. http://maamaaoonny.e//cccchhmhhh

---

Under review as a conference paper at ICLR 2020
Model
Params
BPC
LN HyperNetworks (Ila
2016
27M
1.34
LN HM-LSTM (Chung ct al. 2016
35M
1.32
RHN (Zilly et al.
2017
46M
1.27
Large FS-LSTM 4 (Mujika et al. 2017
47M
1.25
Large mLSTM Krause
2016)
46M
1.24
Transformer (Al-Riou al.
2018)
44M
1.11
Transformer-X0 (Dai
2019
41M
Adaptive-span (Sukhbaatar et al.
2019)
39M
1.02
Explicit Sparse Transformer-X1
41M
1.05
Table 3: Comparison with state-of-the-an results on enwiki8. Explicit Sparse Transformer-X refers to the Transformer with our sparsification method.
Method
En- Vi De-En Training Speed (tokens/s) Inference Speed (tokens/s)
Transformer
30.6
35.3
49K
7.0K
Sparsemax (Martins & Astudillo 2016
31.2
39K
3.0K
Entmax-1.5 (Peters 2019)
30.9
35.6
40K
4.9K
Entmax-alpha (Correia 2019
35.5
13K
0.6K
Proposal
31.1
35.6
48K
6.6K
Table 4: In the Transformer model, the proposed method, top-k selection before softmax is faster than previous sparse attention methods and is comparable in terms of BLEUU scores.
DISCUSSION
In this section, we performed several analyses for further discussion of Explicit Sparse Transformer. First, compare the proposed method of topk selection before softmax with previous sparse attention method including various variants of sparsemax (Martins & Astudillo 2016 Correia et al.) 2019: Peters al.) 2019). Second, discuss about the selection of the value of k. Third, demonstrate that the top-k sparse attention method helps training. In the end, we conducted series of qualitative analyses to visualize proposed sparse attention in Transformer.
5.1 COMPARISON WITH OTHER SPARSE ATTENTION METHODS
We compare the performance and speed our method with the previous sparse attention methods on the basis of strong implemented transformer baseline. The training and inference speed are reported on the platform of Pytorch and IWSLT 2014 De-En translation dataset, the batch size for inference set to 128 in terms of sentence and half precision training(FP-16) is applied.
As we can see from Table 4 the proposed sparse attention method achieve the comparable results as previous sparse attention methods, but the training and testing speed is 2x faster than sparsemax and 10x faster than Entmax-alpp -alpha during the inference. This is due to the fact that our method does not introduce too much computation for calculating sparse attention scores.
The other group of sparse attention methods of adding local attention constraints into attention (Child al. 2019 Sukhbaatar al. 2019). do not show performance neural machine translation, do not compare them in Table
We borrow the implementation of Entmax1.5 in Tensorflow from https::/ htttp:://githbb com/ deep-spin/ entmax sptn/entmax and the implementation of Sparsemax, Entmax-1.5, Entmax alpha Pytorch from https: //gstt . github...... com/ hoom//ustheurissttee weteurris... ic/ 6016 7 29922158 6be 522 7e3cbal We have not found reliable Tensorflow implementation of sparsemax and entmax entmax-alpha in the transformer (we tried to apply the official implementation of sparsemax in Tensorflow to tensor2tensor, but it reports loss of NaN.)

---

Under review as a conference paper at ICLR 2020
IWSLT En-Vi
IWSLT De-En
27.7
36.8
36.7
36.6
27.5
36.5
27.4
27.3
36.3
27.2
36.2
16
32
Inf
16
32
Inf
Figure 3: Analyse the value K on IWSLT En-V Vi and De-En datasets. "inf" denotes the special case of the Explicit Sparse Transformer where all positions may be attended, same as the origin Transformer.
Task
Base
T
T&P
En-Vi (BLEU)
27.4
27.7
27.8
Table 5: Results of the ablation study of the sparsification at different phases the En- Vi test set. "Base" denotes vanilla Transformer. "T" denotes only adding the sparsification in the training phase, and "T&P" denotes adding at both phases the implementation of Explicit Sparse Transformer does.
5.2 HOW TO SELECT A PROPER
The natural question of how to choose the optimal comes with the proposed method. We compare the effect of the value of at exponential scales. We perform experiments on En-Vi and De-En from the valid set.
3 different initializations for each value of K, and report the mean BLEU scores The figure shows that regardless the value of on the En- Vi dataset, the model performance generally rises first and then falls as A: increases. Under the setting of the k E (4, 8, 16, 32), setting the value of to 8 achieves consistent improvements over the
5.3 DO THE PROPOSED SPARSE ATTENTION METHOD HELPS TRAINING?
We surprised find that only adding the sparsification in the training phase can also bring improvement in the performance. We experiment this idea on IWSLT En-Vi and report the results on the valid set in Table The improvement of 0.3 BLEU scores shows that vanilla Transformer may be overparameterized and the sparsification encourages the simplification of the model.
5.4 Do THE EXPLICIT SPARSE TRANSFORMER ATTEND BETTER?
To perform a thorough evaluation of our Explicit Sparse Transformer, we conducted case study and visualize the attention distributions of our model and the bascline for further comparison. Specifically, we conducted the analysis on the test set En- Vi, and randomly selected sample pair of attention visualization of both models.
The visualization of the context attention of the decoder's bottom layer in Figure 4(a) The attention distribution of the left figure is fairly disperse. On the contrary, the right figure shows that the sparse attention can choose to focus only on several positions so that the model can be forced to stay focused. For example, when generating the phrase "for thinking about my heart"(WWrr--te" word translation
7

---

Under review as a conference paper at ICLR 2020
Geart
Seac.
140
(a) Attention of the bottom layer
(b) Attention the top layer
Figure 4: Figure ((a) is the attention visualization of Transformer and Figure 4(b) that of the Explicit Sparse Transformer. The red box shows that the attentions in vanilla Transformer at most steps are concentrated on the last token of the context.
from Vietnamese), the generated word cannot be aligned to the corresponding words. As Explicit Sparse Transformer, when generating the phrase "with all my heart", the attention can focus on the corresponding positions with strong confidence.
The visualization of the decoder's top layer is shown in Figure 4(b) From the figure, the context attention the top layer of the vanilla Transformer decoder suffers from focusing on the last source token. This is common behavior of the attention in vanilla Transformer. Such attention with wrong aligmment cannot sufficiently extract enough relevant source-side information for the generation. In contrast, Explicit Sparse Transformer, with simple modification on the vanilla version, does not suffer from this problem, but instead focuses on the relevant sections of the source context. The figure on the right demonstrating the attention distribution of Explicit Sparse Transformer shows that our proposed attention in the model is able to perform accurate alignment.
RELATED WORK
Attention mechanism has demonstrated outstanding performances in a number of neural-network- based methods, and it has been focus in the NLP studies (Bahdanau al. (2014). A number of
studies are proposed to enhance the effects of attention mechanism (Luong et al. [013 Vaswani et al.) 2017 Ke al. 2018). Luong al. (2015) propose local attention and Yang al. (0118) propose local attention for self-attention, Xu et al. (2015) propose hard attention that pays discrete attention in image captioning. Chandar al, (2016) propose combination soft attention with hard attention to construct hierarchical memory network. Lin et al. (2018) propose a temperature mechanism to change the softness of attention distribution. Shen et al. (2018) propose an attention which can select a small proportion for focusing. It is trained by reinforcement learning algorithms (Williams) 1992).
In terms of memory networks, Rae al. (2016) propose to sparse access memory
Child et al. (2019) recently propose to use local attention and block attention to sparsify the trans- former. Our approach differs from them in that our method does not need block sentences and still capture long distance dependencies. Besides, we demonstrate the importance of Explicit Sparse Trans- former in sequence to sequence learning. Although the variants of sparsemax (Martins & Astudillo 2016 Correia et al. 2019 Peters et al. 2019) improve in machine translation tasks, we empirically demonstrate in 5.1 that our method introduces less computation in the standard transformer and is much faster than those sparse attention methods GPUs.
7 CONCLUSION
In this paper, we propose novel model called Explicit Sparse Transformer. Explicit Sparse Trans- former is able to make the attention vanilla Transformer more concentrated on the most contributive components. Extensive experiments show that Explicit Sparse Transformer outperforms vanilla Trans- former in three different NLP tasks. We conducted a series of qualitative analyses to investigate the reasons why Explicit Sparse Transformer outperforms the vanilla Transformer. Furthermore, find
8

---

Under review as a conference paper at ICLR 2020
an obvious problem of the attention at the top layer of the vanilla Transformer, and Explicit Sparse Transformer can alleviate this problem effectively with improved alignment effects.
REFERENCES
Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. Weighted transformer network for machine translation. CoRR, abs/1711.02132. 2017.
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. Character-level language modeling with deeper self-attention. arXiv preprint arXiv: 04444, 2018.
Peter Anderson, Xiaodong Hc, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6077-6086, 2018.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. CoRR, 0os//149.0077, 2014.
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron C. Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In 5th Interna- fional Conference on Learning Representations, ICLR 2017, 2017.
Loris Bazzani, Tobias Domhan, and Felix Hieber. Image captioning as neural machine translation task in sockeye. arXiv preprint 1rXiv:1810.04110 2018a.
Loris Bazzani, Tobias Domhan, and Felix Hieber. Image captioning as neural machine translation task in SOCKEYE. CoRR, abs/ 18 0.04101, 2018b.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign. iwslt 2014. In Proceedings of the International Workshop on Spoken Language Translation, Hanoi, Vietnam, 2014.
Sarath Chandar, Sungjin Ahn, Hugo Larochelle, Pascal Vincent, Gerald Tesauro, and Yoshua Bengio. Hierarchical memory networks. CoRR, abs/1605.07427, 2016.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. The best of both worlds: Combining recent advances in neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, pp. 76-86, 2018
X. Chen, H. Fang, TY Lin, R. Vedantam, S. Gupta, P. Dollr, and C. L. Zitnick. Microsoft coco captions: Data collection and evaluation server. 3xXiv:5504.0032. 2015a.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1 abs/1504.00025.. 2015b.
Rewon Child, Scott Gray, Alee Radford, and llya Sutskever. Generating long sequences with sparse transformers, 2019.
Junyoung Chung. Çaglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks. In Proceedings of the 32nd International Conference on Machine Learning, ICML 2015, pp. 2067-2075, 2015.
Junyoung Chung, Sungjin Ahn, and Yoshua Bengio. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv: 1609.66000000 2016.
Gonalo M. Correia, Vlad Niculae, and Andr F. T. Martins. Adaptively sparse transformers, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. Transformer-xi Attentive language models beyond a fixed-length context. arXiv preprint arXiv: 190100280. 2019.

---

Under review as a conference paper at ICLR 2020
Mostafa Dehghani, Stephan Gouws, Oriol Vinyals, Jakob Uszkoreit, and Lukasz Kaiser. Universal transformers. CoRR, abs//807.038113 2018.
Yuntian Deng. Yoon Kim, Justin Chiu, Demi Guo, and Alexander M. Rush. Latent alignment and variational attention. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, pp. 9735-9747, 2018.
Michael J. Denkowski and Alon Lavie. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, pp. 376-380, 2014.
Jacob Devlin, Ming- Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint vrXiv:1810.04800 2018.
Sergey Edunov, Myle Ott, Michael Auli, David Grangier, and Mare' Aurelio Ranzato. Classical structured prediction losses for sequence sequence learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2018 (Long Papers), pp. 355-364, 2018.
Jiangtao Feng, Lingpeng Kong, Po-Sen Huang. Chong Wang, Da Huang. Jiayuan Mao, Kan Qiao, and Dengyong Zhou. Neural phrase-to-phras machine translation. CoRR, abs/181 111 102172, 2018.
Jos A. R. Fonollosa, Noe Casas, and Marta R. Costa-juss. Joint source-target self attention with locality constraints, 2019.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. Convolutional sequence to sequence learning. In ICML 2017, volume 70 of Proceedings of Machine Learning Research, pp. 1243-1252. PMLR, 2017.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. arXiv preprint arXiv: 2016.
Tianyu Hc, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie- Yan Liu. Layer-wise coordination between encoder and decoder for neural machine translation. In Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Gamett (eds.), Advances in Neural Information Processing Systems 31, pp. 7944-7954. Curran Associates, Inc., 2018.
Po-Sen Huang, Chong Wang, Sitao Huang, Dengyong Zhou, and Li Deng. Towards neural phrase- based machine translation. arXiv preprint arXivv1706.055655 2017.
Andrej Karpathy and Fei-Fei Li. Deep visual-semantic alignments for generating image descriptions. In CVPR 2015, pp. 3128-3137. IEEE Computer Society, 2015.
Nan Rosemary Ke, Anirudh Goyal, Olexa Bilaniuk, Jonathan Binas, Michael C. Mozer, Chris Pal, and Yoshua Bengio. Sparse attentive backtracking: Temporal credit assignment through reminding. In NeurlPS 2018, pp. 7651-7662, 2018
Ben Krause, Liang Lu, lain Murray, and Steve Renals. Multiplicative 1stm for sequence modelling. arXiv preprint arXiv: 1609.07959, 2016.
Junyang Lin, Xu Sun, Xuancheng Ren, Muyu Li, and Qi Su. Learning when to concentrate or divert attention: Self-adaptive attention temperature for neural machine translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp.
2018.
Jiasen Lu, Caiming Xiong. Devi Parikh, and Richard Socher. Knowing when to look: Adaptive attention via visual sentinel for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 375-383, 2017.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. Neural baby talk. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7219-7228, 2018
Thang Luong, Hieu Pham, and Christopher D. Manning. Effective approaches to attention-base neural machine translation. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pp- 1412-1421, 2015.
10

---

Under review as a conference paper at ICLR 2020
André F. T. Martins and Ramón Fernández Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In ICML 2016, pp. 1614-1623, 2016.
Asier Mujika, Florian Meier, and Angelika Steger. Fast-slow recurrent neural networks. In Advances in Neural Information Processing Systems, pp. 5915-5924, 2017.
Myle Ott. Sergey Edunov, David Grangier, and Michael Auli. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, WMT 2018, Belgium, Brussels, October 31- November 1, 2018, pp. 1-9, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In ACL. 2002, pp. 311-318, 2002.
Ben Peters, Vlad Niculae, and Andr F. T. Martins. Sparse sequence- to of the 57th Annual Meeting of the Association for Computational I Linguistics, 2019. doi: 10.18653) v1/p19-1 1---1444 146. URL /xx. doi. www.orgg...... wat.l1863////// 1663//////..1133////////3/3//3
to-uunee----sqquenee sequence models. Proceedings
Jack W Rae, Jonathan J Hunt, Tim Harley, Ivo Danihelka, Andrew Senior, Greg Wayne, Alex Graves, and Timothy Lillicrap. Scaling memory-angmented neural networks with sparse reads and writes, 2016.
Steven Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7008-7024, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguisticss, ACL 2016, 2016.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 2 (Short Papers), pp. 464 468, 2018.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, and Chengqi Zhang. Reinforced self-attention network: hybrid of hard and soft attention for sequence modeling. In IJCAI 2018, pp. 4345-4352, 2018.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019, doi: 10. 10.18653/v1/p191000 URL http:/ //tx://xxxx doi wtt.ooo............3...33/
019-1032
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS 2017, pp. 6000-6010, 2017.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR 2015, pp. 35566-475,,, 2015.
Ronald Williams. Simple statistical folliinn algorithms for connectionist reinforcement learning. In Machine Learning. pp. 229-256. 1992.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv: 2019.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron C. Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In ICML 2015, pp. 4048-2057, 2015.
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 4449 4458, 2018.
11

---

Under review as a conference paper at ICLR 2020
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. CoRR, 0oo//9911.03322 2019.
Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. Recurrent highway networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 4189-4198, JMLR. org, 2017.
A APPENDIX
A.1 BACKGROUND
A.1.1 ATTENTION MECHANISM
Bahdanau al. 2014)) first introduced the attention mechanism to learn the alignment between the side context and the source-side context, and Luong et al. (2015) formulated several versions for local and global attention. In general, the attention mechanism maps query and a key-value pair to output. The attention score function and softmax normalization can turn the query Q and the key into a distribution a. Following the distribution a, the attention mechanism computes the expectation of the value V and finally generates the output C.
Take the original attention mechanism in NMT as an example. Both key K E R2xd and value V E Raxd are the sequence of output states from the encoder. Query Q Rm is the sequence of output states from the decoder, where m is the length of Q, n is the length of K and V, and d is the dimension of the states. Thus, the attention mechanism is formulated as:
softmax(f(Q K))V
(5)
where f refers to the attention score computation.
A.1.2 TRANSFORMER
Transformer Vaswani et al. 2017), which is fully based the attention mechanism, demonstrates the state-of-the-art performances in series of natural language generation tasks. Specifically, we focus on self-attention and multi-head attention.
The ideology of self-attention is, as the name implies, the attention over the context itself. In the implementation, the query Q key K and value V are the linear transformation of the input x, so that Q Woz, K = x and V = Wvz where Wo. WK and Wy are learnable parameters. Therefore, the computation can be formulated below:
QKT Vd
C = softmax
V
(6)
where d refers to the dimension of the states.
The aforementioned mechanism can be regarded as the unihead attention. As to the multi-head attention, the attention computation is separated into g heads (namely 8 for basic model and 16 for large model the common practice). Thus multiple parts of the inputs can be computed individually. For the i-th head, the output be computed in the following formula:
(Q000)
(iti) = softmax
V(i
(7)
Vdh
where (00 refers to the output of the head, Q(i,, K0 and v() are the query, key and value of the head, and de refers to the size of each head (dN d/g). Finally, the output of each head are concatenated for the output:
C C= (C
c(
_(O).... ..... c
(8)
In common practice, C' is sent through linear transformation with weight matrix W. for the final output of multi-head attention.
However, soft attention assign weights to a lot words that less relevent to the query. Therefore, in order to improve concentration in attention for effective information extraction, we study the problem of sparse attention in Transformer and propose our model Explicit Sparse Transformer.
12

---

Under review as a conference paper at ICLR 2020
A.2 EXPERIMENTAL DETAILS
We use the default setting in Vaswani et al. (2017) for the implementation of our proposed Explicit Sparse Transformer. The hyper parameters including beam size and training steps tuned on the valid set.
Neural Machine Translation Training For En-Vi translation, we use default scripts and hyper- to preprocess, train and evaluate our model. We use the default scripts of fairsed v0.6.1 to preprocess the De-En and En-De dataset. We train the model on the En- Vi dataset for 35K steps with batch size of 4K. For IWSLT 2015 5 De-En dataset, batch size also set to 4K, we update the model every steps and train the model for 90ppochs. For WMT 2014 En-De dataset,
parameter setting of tensor2tenso
train the model for 72 epochs on 4 GPUs with update frequency of 32 and batch size 3584. We train all models on a single RTX2080TI for two small IWSLT datasets and on single machine of 4 RTX TITAN for WMT14 En-De. In order to reduce the impact of random initialization, we perform experiments with three different initializations for models and report the highest for small datasets.
Evaluation We use case-sensitive tokenized BLEU score (Papineni et al. 2002) for the evaluation of WMT14 En-De, and use case-insensitive BLEU for that of IWSLT 2015 En-Vi and IWSLT 2014 De-En following [in et al.) (008)). Same as Vaswani et al. (2017). compound splitting is used for WMT 14 En-De. For WMT 14 En-De and IWSLT 2014 De-En, we save checkpoints every epoch and average last 10 checkpoints every 5 epochs. We select the averaged checkpoint with best valid BLEU and report its BLEU score on the test set. For IWSLT 2015 En-Vi, we save checkpoints every 600 seconds and average last 20 checkpoints.
still use the default setting of Transformer for training our proposed Explicit Sparse Transformer. We report the standard automatic evaluation metrics with the help of the COCO captioning evaluation toolki (Chen et 2015b), which includes the commonly-use evaluation metrics, BLEU-4 Papineni et al. (2002) M METEOR Denkowski & Lavie (2014), and CIDEr Vedantam
Image Captioning
et al. 2015).
Language Models We follow Dai al. (2019) and use their implementation for Explicit Sparse Transformer. Following the previous work (Chung et al. 2015 Dai et al. 2019), we use BPC (F(log2 + 1.)t))). standing for the average number of Bits-Per-Chhen Per-Characcr,,r, for evaluation. Lower BPC refers to better performance, As to the model implementation, we implement Explicit Sparse Transformer-XL, which is based the base version of Transformer-XLL [ Transformer-XX. a model based on Transformer but has better capability of representing long sequences.
A.3 THE BACK-PROPAGATION PROCESS OF TOP-K SELECTION
The masking function M(:..) is illustrated as follow:
M(P, k)i { Poj if Psj 2 t (k-th largest value of row i) [-0
(9)
if Pij < la (k-th largest value of row i)
Denote M M(P,k) k). We regard & as constants. When back-propagating,
0Mg (i or +
(10)
0Pи
0Mg
[1
if P.j 2 t (k-th largest value of row 2) if P.j (k-th largest value of row i)
(11)
OP
0
'https:/gitthub. "herp://gghubb.om/tttssffroovo com//ensoorfoo///nssooeeetaa
hhtrp://ggtuuh.om/yyrrr//aaaaaie com/pytoocch/faireq
"https://itthub https//github.com/// httpss//gghhhh.omttiiiiccoooo com/tylin/coco-captioo
Due to our limited resources (TPU), we did not implement the hig version of Explicit Sparse Transformer-
XL.
13

---

Under review as a conference paper at ICLR 2020
The next step after top-k selection is normalization:
A = softmax(M(P,.) k))
(12)
where A refers to the normalized scores.
When backpropagating.
la
AAi
0M
(13)
0P
mM OPkt
m=1 n=1
aAi, 0MR
(14)
0M
0Pk
AAi
if Pij (k-thh largest value of row i)
0Mи
(15)
0 Pt < tq (k-tth largest value of row 2)
The softmax function is evidently differentiable, therefore, we have calculated the gradient involved in top-k selection.
A.4 IMPLEMENTATION
Figure[Jshowe shows the code for the idea case of single head self-attention, the proposed method is easy to implement and plug in the successful Transformer model.
14

---

Under review as a conference paper at ICLR 2020
import torch
import torch. functional def sparse_ dot dot_ppoduucttteeeecooooee (key, value, quers, k-0);
Key module of Sparse Transformer, same time efficiency as the origin self attentic Compute the context vector and the attention vectors.
Args:
key ('FloatTensor') :
set "key_len"
10
key vectors '[batch, key_ dim] value (FloatTensor'): ): of lan value vectors "[batch, key_len, dim] query (FFloatTensor') of 'query_len" query vectors "[batch, query_len, dim] ('int') select kt posi tions
12
14
15
16
Re turns:
17
FFloatTensor', "Flaatennoor) )
18
output context rectors [batch, query, Ien, dim]"
19
of the attention vectors [batch, query_ Ien, key, len]"
20
21
1) Calculate self attention scores.
22
scores torch. matmul (query, key, transpose bs, query_ len, key,
23
.2) Compute the sparse attention mask and mask out the attention on values small magni
25
if k >> key. 0 [1]: k key, size [1]
27
if k:
28
torch. tspores k)
29
print(ralue))
30
vk vi:, -1]. unsqueeze (2) expand_assssaaeeeeeee (scorees)
31
mask torch. 1t (scores, scores scores. masked_ fill (mask_k, -1e18)
32
33
Normalize and compute context vectors.
34
attn F. (soome)
35
context torch. matmul (attn, value)
36
return context, attn
Figure 5: Code for the main idea in Pytorch
15


Reviews:
[{"review_id": "SkxkjiCatr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes \"sparse self-attention\", where only top K activations are kept in the softmax. The resulting transformer model is applied to NMT, image caption generation and language modeling, where it outperformed a vanilla Transformer model.\n\nIn general, the idea is quite simple and easy to implement. It doesn't add any computational or memory cost. The paper is well written and easy to read. The diverse experimental results show that it brings an improvement. And I think this can be combined with other improvements of Transformer.\n\nHowever, there are quite many baselines are missing from the tables. The sota on De-En is actually 35.7 by Fonollosa et.al. On enwik8, Transformer XL is not the best medium sized model as the authors claimed. See below:\n\nNTM En-De: \n- Wu et.al. Pay Less Attention with Lightweight and Dynamic Convolutions, 2019\n- Ott et.al. Scaling Neural Machine Translation, 2018\nNTM En-Vi: \n- Wang et.al. SwitchOut: an Efficient Data Augmentation Algorithm for Neural Machine Translation, 2018 \nNTM De-En: \n- Wu et.al. Pay Less Attention with Lightweight and Dynamic Convolutions, 2019\n- Fonollosa et.al. Joint Source-Target Self Attention with Locality Constraints, 2019\n- He et.al. Layer-Wise Coordination between Encoder and Decoder for Neural Machine Translation, 2018\nLM Enwik8:\n- Sukhbaatar et.al, Adaptive Attention Span in Transformers, 2019\n\nOther comments:\n- More experimental details are needed. What is the value K? How different K values affect performance? What is the number of parameters of NMT models.\n- The claim \"top layer of the vanilla Transformer focuses on the end position of the text\" can't be true generally. Probably only true for a certain task. \n- Where the numbers in Figure 1 come from? Is it a single attention head or average of all?\n- Page 4, \"the high are ...\" probably typo?\n- The related work is missing \"Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes\" by Rae et.al., which also uses sparse attention.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "BkehHdjsFS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "1. What is the specific question/problem tackled by the paper?\n\nThe authors tackle the problem of sparse attention for various generative modeling tasks such as machine translation and image captioning. The main motivation behind studying this problem is the premise that sparse varieties of attention might generalize better than full attention. The authors propose a sparse attention mechanism based on the top-k selection where all attention values in a row are dropped if they are not higher than the k^{th} largest item in the row. Since this is a non-differentiable operation the authors propose to train this model by setting the gradients of the non-selected items to 0. The authors report results on machine translation, language modeling and image captioning.\n\n2. Is the approach well motivated, including being well-placed in the literature?\n\nIn my view the main reasons to study sparse variants of attention are either 1) scale to sequences longer than are possible with full attention (this is e.g., the motivation behind [1]) or 2) generalize better than full attention. The motivation of this work seems to be the latter as the authors claim improvements in terms of performance over full attention. The authors cite prior work on sparse attention mechanisms.\n\n3. Does the paper support the claims? This includes determining if results, whether theoretical or empirical, are correct and if they are scientifically rigorous.\n\nThe authors report good results on machine translation, showing that their sparse attention method improves performance on En-De to 29.4 BLEU, on De-En to 35.6 BLEU and on En-Vi to 31.1 BLEU, improving on full attention baselines. However, the authors have not submitted code for reproducing their results. The authors also do not report what choice of k is used for the top-k operation and how they made their choice of the optimal k? The paper would be well served by more ablation experiments demonstrating what the impact the choice of k has on the model performance. For example, I would expect to be able to reproduce original Transformer results using k = maximum sequence length. \n\nI am also not fully clear about how gradients are propagated through the top-k operation. It seems that if an index is not selected (i.e. it's attention value is smaller than top-k) it's gradient is set to 0. However, this seems problematic - for e.g., in the initial stages an important item might have a low attention value due to random initialization and might not make it to the top-k. Because of the way gradients are propagated it will not receive any gradient, and therefore will not be incentivized to increase its value. This doesn't seem like a good solution to me.\n\nSince the paper is mainly an empirical work, it would be improved by open-sourcing anonymized code so that it's results and claims may be verified. It would also be improved in more ablation experiments or explanations in what the optimal choice of k should be for the top-k and how that affects the results. \n\n[1] Generating Long Sequences with Sparse Transformers by Child et al (https://arxiv.org/abs/1904.10509)\n\n ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "B1x0T9sPtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "CONTRIBUTIONS:\nC1. Sparse Transformer: A modification of the Transformer, limiting attention to the top-k locations. (That is a complete statement of the proposed model.)\nC2. Experiments showing that, quantitatively, the Sparse Transformer out-performs the standard Transformer on translation, language modeling, and image captioning.\nC3. Experiments showing that, qualitatively, in translation, when generating a target word, the Sparse Transformer better focusses attention on the aligned source word \n\nRATING: Reject\n\nREASONS FOR RATING (SUMMARY). The innovativeness seems low given the several previous proposals for sparse attention, the results are not dramatic enough to compensate for the lack of originality, and the comparison to other models is wanting.\n\nREVIEW\n\nStrengths: The paper is clearly written. The question of whether the Transformer’s attention is too diffuse is of interest. The proposal is admirably simple. The quantitative metrics include comparison against many alternative models.\n\nWeaknesses: A primary area of deficiency concerns the relation of the proposed model to other proposals for sparse attention: the authors cite 5 of them (and 2 more are cited in the comment by Cui). The paper should clearly identify the differences between the proposed model and earlier models: it does not discuss this at all. The deficiencies in these previous models should be clearly stated and demonstrated:  they are only described as “either restricted range of attention or training difficulty” (Sec 6). A rationale for why the proposal can be expected to remedy these deficiencies should be stated clearly: it is not stated at all. Experimental demonstration that the proposed innovation actually remedies the identified deficiencies should be provided, but is not.\n\nA proposal to use a top-k filter immediately raises the question of the value of k. This is not discussed at all. In particular, no empirical results are given concerning the sensitivity of the reported successes to choosing the correct value for k. We are only told that “k is usually a small number such as 5 or 10” (Sec 3). The experimental details in the appendix do not even state the value of k used in the models reported.\n\nIt is an interesting discovery that in the translation task, attention at the top layer of the standard Transformer is strongly focused on the end of the input. This is described as an “obvious problem” (Sec 7). But it can’t obviously be a problem because the performance of the standard Transformer is only very slightly lower than that of the Sparse Transformer: if anything is obvious, it is that processing in the standard Transformer packs a lot of information into its final encoding of the end of the input string, which functions rather like an encoding of the entire sentence.\n\nPresumably, the experimental results reported are those from a single model, since we are not told otherwise. There should be multiple tests of the models with different random initializations, with the means and variances of measures reported. It is possible, however, that limitations of computational resources made that infeasible, although the Appendix seems to indicate that no hyperparameter tuning was done, which greatly reduces computational cost.\n\nCOMMENTS FOR IMPROVEMENT, NOT RELEVANT TO RATING DECISION\n\nAlthough the tiny sample of visualized attention weights provided is useful, a large-scale quantitative assessment of a main claim concerning translation might well be possible: that attention is in fact concentrated on the aligned word might be testable using an aligned bilingual corpus or perhaps an existing forced aligner could be used.\n\nMuch space could be saved: it is not necessary to review the standard Transformer, and the modification proposed is so simple that it can be precisely stated in one sentence (see C1 above): the entire page taken up by Sec. 3 is unnecessary, as it adds only implementation details.\n\nErrors that took more than a moment to mentally correct, all on p. 12:\n\nThe definition of the BPC should be E[log P(x(t+1) | h(t))]: all parentheses are missing\n“regrad” should be “regard”\n“derivative” should be “differentiable” in the final sentence", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a variant of Sparse Transformer where only top K activations are kept in the softmax. The resulting transformer model is applied to NMT, image caption generation and language modeling, where it outperformed a vanilla Transformer.

While the proposed idea is simple, easy to implement, and it does not add additional computational or memory cost, the reviewers raised several concerns in the discussion phase, including: several baselines missing from the tables; incomplete experimental details; incorrect/misleading selection of best performing model in tables of results (e.g. In Table 1, the authors boldface their results on En-De (29.4) and De-En (35.6) but in fact, the best performance on these is achieved by competing models, respectively 29.7 and 35.7. The caption claims their model "achieves the state-of-the-art performances in En-Vi and De-En" but this is not true for De-En (albeit by 0.1). In Table 3, they boldface their result of 1.05 but the best result is 1.02; the text says their model beats the Transf-XL "with an advantage" (of 0.01) but do not point out that the advantage of Adaptive-span over their model is 3 times as large (0.03)).

This prevents me from recommending acceptance of this paper in its current form. I strongly encourage the authors to address these concerns in a future submission.
Decision: Reject