SlowGuess/ACL_2020_OCR huggingface dataset:
Encoder, Decoder and Cross Attentions. Our proposed methods can be readily applied to the any of the attention layers in the Transformer framework. We could also selectively apply our methods to different layers in the encoder and decoder.
6593
In our initial experiments on WMT'14 English-German development set, we observed that the following settings provide more promising performance gains.
First, encoder self-attention layers benefit most from additive window attention, while decoder self-attention layers prefer multiplicative attention. This shows that the global attention component is more useful when the key sequence is provided entirely in the encoder, while less useful when only the fragmented key sequence (past keys) is visible in the decoder. Second, the above argument is further reinforced as we found that cross-attention layers also prefer additive window attention, where the entire source sequence is available. Third, cross-attention works better with segment-based masking, which provides smoothness and facilitates phrase (n-gram) based translations.
Lower-layer Local Attentions. It has been shown that deep neural models learn simple word features and local syntax in the lower layers, while higher layers learn more complex context-dependent aspects of word semantics. Belinkov et al. (2017) show this on NMT models, while Peters et al. (2018) and Jawahar et al. (2019) show this on representation learning with ELMo and BERT respectively. In other words, local contextual information can still be derived in higher layers with the standard global attention. As such, we propose to apply our dynamic window attention methods only to the first 3 layers of the Transformer network, leaving the top 3 layers intact. Our diverse experiments in the following section support this setup as it offers substantial improvements, whereas using local attention in higher layers does not show gains, but rather increases model parameters.
In this section, we present the training settings, experimental results and analysis of our models in comparison with the baselines on machine translation (MT), sentiment analysis, subject verb agreement and language modeling (LM) tasks.
We trained our models on the standard WMT'16 English-German (En-De) and WMT'14 English-French (En-Fr) datasets containing about 4.5 and 36 million sentence pairs, respectively. For validation (development) purposes, we used newstest2013 for En-De and a random split from the
training set for En-Fr. All translation tasks were evaluated against their respective newstest2014 test sets, in case-sensitive tokenized BLEU. We used byte-pair encoding (Sennrich et al., 2016) with shared source-target vocabularies of 32,768 and 40,000 sub-words for En-De and En-Fr translation tasks, respectively. We compare our models with three strong baselines: (i) Transformer Base (Vaswani et al., 2017), (ii) Transformer Base with Relative Position (Shaw et al., 2018), and (iii) Transformer Base with Localness Modeling (Yang et al., 2018). To ensure a fair comparison, we trained our models and the baselines with the following training setup.
Training Setup. We followed model specifications in (Vaswani et al., 2017) and optimization settings in (Ott et al., 2018), with some minor modifications. Specifically, we used word embeddings of dimension 512, feedforward layers with inner dimension 2048, and multi-headed attentions with 8 heads. We trained our models on a single physical GPU but replicated the 8-GPU setup following the gradient aggregation method proposed by Ott et al. (2018). We trained the models for 200,000 updates for En-De and 150,000 updates for En-Fr translation tasks. Finally, we averaged the last 5 checkpoints to obtain the final models for evaluation. The segment size  $b$  in the segment-based masking method was set to 5.
Translation Results. We report our translation results in Table 1; Enc(AW) indicates the use of additive window (AW) attention in the encoder, Dec(MW) indicates the use of multiplicative window (MW) attention in the decoder, and Cr(AW,Seg) indicates the use of additive window attention with segment-based masking for cross-attention. The attention module that is not specified in our naming convention uses the default token-based global attention in the Transformer. For example, Enc(AW)-Dec(MW) refers to the model that uses AW attention in the encoder, MW attention in the decoder and the default global attention for cross attention.
We notice that despite a minor increase in the number of parameters, applying our attentions in the encoder and decoder offers about 0.7 and 1.0 BLEU improvements in En-De and En-Fr translation tasks respectively, compared to the
3We did not tune  $b$ ; tuning  $b$  might improve the results further.
6594
Model	#-params	En-De	En-Fr
Vaswani et al. (2017)	63M	27.46	39.21
Shaw et al. (2018)	63M	27.56	39.37
Yang et al. (2018)	63M	27.62	39.47
Our Models
Enc(AW)-Dec(MW)	68M	28.11	40.24
Cr(AW, Seg)	65M	28.13	40.06
Enc(AW)-Cr(AW,Seg)-Dec(MW)	73M	28.25	40.32
Table 1: BLEU scores for different models in WMT'14 English-German and English-French translation tasks.
Method	Module	Full (6 layers)	Partial (3 layers)
Transformer	-	27.46	-
AW	Encoder	27.77	27.90
MW	Encoder	27.25	27.40
AW	Decoder	27.73	27.85
MW	Decoder	27.88	28.04
AW	Cross	27.78	27.97
MW	Cross	27.58	27.79
Transformer base (Vaswani et al., 2017). Our model with the segment-based additive method for cross attention achieves a similar performance. We observe further improvements as we apply our attentions in all the attention modules of the Transformer. Specifically, our model Enc(AW)-Cr(AW,Seg)-Dec(MW) achieves 28.25 and 40.32 BLEU in En-De and En-Fr translation tasks, outperforming Transformer base with localness (Yang et al., 2018) by 0.63 and 0.85 BLEU, respectively.
To verify our modeling decisions, we performed an ablation study in the WMT'14 En-De translation task. In particular, we evaluated  $(i)$  the impact of applying our differentiable window attentions in all layers vs. only in certain lower layers of the Transformer network,  $(ii)$  which window attention methods (additive or multiplicative) are suitable particularly for the encoder/decoder self-attention and cross-attention, and  $(iii)$  the impact of segment-based masking in different attention modules.  $(iv)$  training efficiency and performance of our best model with the similar models. Plus, to further interpret our window-based attention, we also provide the local window visualization.
Full vs. Partial. Table 2 shows BLEU scores for the Transformer models that employ our window
Table 2: Evaluation of Additive Window (AW) and Multiplicative Window (MW) attentions in encoder/decoder self attention and cross attention for full vs. partial settings.
Model	Token-based	Segment-based
Cr(AW)	27.97	28.13
Enc(AW)-Dec(MW)	28.11	27.91
Table 3: BLEU scores for token- and segment-based masking in cross attention and encoder self-attention. The decoder self-attention always uses token-based masking.
based attentions in all 6 layers (Full) vs. only in the first 3 layers (Partial), as well as the methods used in different attention modules (encoder/decoder self-attention, cross-attention). We can see that almost all the models with window-based methods in the first 3 layers outperform those that use them in all 6 layers. This gives the setup significant advantages as it performs not only better in BLEU but also requires less parameters.
The results also show that multiplicative window (MW) attention is preferred in decoder self-attention, while additive window (AW) is more suitable for encoder self-attention and for cross-attention. This suggests that the global context, which is maintained in AW, is more useful when it is entirely available like in encoder self-attention and cross attention. In contrast, incomplete and partially-generated context in decoder self-attention may induce more noise than information, where MW attention renders better performance than AW.
Token- vs. Segment-based. Table 3 compares the results for using token-based vs. segment-based masking methods in different attention modules of the network. Note that it is preferred for decoder self-attention to adopt token-based masking since the decoder cannot point to unfinished segments in autoregressive generation, if it had used segment-based masking. We see that segment-based additive window masking outdoes its token-based counterpart (28.13 vs. 27.97 BLEU) for cross-attention. Meanwhile, for encoder self-attention, token-based masking performs better than segment-based masking by 0.2 BLEU. This suggests that segments (or phrases) represent better translation units than tokens, justifying its performance superiority in cross-lingual attention but not in monolingual (self-attention) encoding.
Speed and Parameters. As shown in table 4, our training efficiency is competitive to the baselines. That is, the training speed for our model is 1.04
(a) Local masking scores  $(M)$ .
(b) Our attention scores.
Figure 3: Visualization of masking scores, and attention scores for our and the original Transformer models.
(c) Transformer attention scores.
Model	#-params	# steps/sec	BLEU
Vaswani et al. (2017)	63M	1.20	27.46
Yang et al. (2018)	63M	1.07	27.62
Vaswani et al. (2017) 7 layers	69M	1.05	27.74
Vaswani et al. (2017) 8 layers	75M	0.99	27.89
Enc(AW)-Cr(AW,Seg)-Dec(MW)	73M	1.04	28.25
steps/sec which is similar to Yang et al. (2018). Besides, our model outperforms the Transformer with 8 layers, which has more parameters. This suggests that our performance gain may not come from additional parameters, but rather from a better inductive bias through the dynamic window attention.
Local Window Visualization. To further interpret our window-based attentions, Figure 3a shows the cross-attention soft masking values  $(\hat{m}_q)$  on the source tokens for each target token in an En-Fr test sample assigned by our Enc(AW)-Cr(AW,Seg)-Dec(MW) model. The darker the score, the higher the attention is from a target token to a source token. We can see the relevant subwords are captured by the attentions quite well, which promotes ngram-level alignments. For instance, the mask  $(\hat{m}_q)$  guides the model to evenly distribute attention scores on sub-words "Co@@" and "en" (Fig. 3b), while standard attention is biased towards "Co@@" (Fig. 3c). Similar phenomenon can be seen for "Bro@@" and "thers" (towards "frères").
5.3 Text Classification
We evaluate our models on the Stanford Sentiment Treebank (SST) (Socher et al., 2013), IMDB sentiment analysis (Maas et al., 2011) and Subject-Verb Aggreement (SVA) (Linzen et al., 2016) tasks. We compare our attention methods (incorporated into the Transformer encoder) with the encoders of Vaswani et al. (2017), Shaw et al. (2018) and Yang et al. (2018).
Table 4: Training efficiency and size of similar models
Model	STT	IMDB	SVA
Vaswani et al. (2017)	79.36	83.65	94.48
Shaw et al. (2018)	79.73	84.61	95.27
Yang et al. (2018)	79.24	84.13	95.00
Enc (MW)	79.70	85.09	95.95
Enc (AW)	82.13	87.98	96.19
Table 5: Classification accuracy on on Stanford Sentiment Treebank (SST) and IMDB sentiment analysis and Subject-Verb Agreement(SVA) tasks.
Training Setup. As the datasets are quite small compared to the MT datasets, we used tiny versions of our models as well as the baselines. Specifically, the models consist of a 2-layer Transformer encoder with 4 attention heads, 128 hidden dimensions and 512 feedforward inner dimensions. In these experiments, our attention methods are applied only to the first layer of the network. We trained for 3,000, 10,000 and 10,000 updates for SST, IMDB and SVA tasks, respectively on a single GPU machine.
Results. Table 5 shows the results. Our multiplicative window approach (Enc (MW)) achieves up to  $79.7\%$ ,  $85.1\%$  and  $95.95\%$  accuracy in SST, IMDB and SVA, exceeding Transformer (Vaswani et al., 2017) by  $0.4\%$ ,  $1.35\%$  and  $1.47\%$ , respectively. Our additive window attention (Enc (AW)) renders even more improvements. Specifically, it outperforms Transformer with relative position (Shaw et al. 2018) by  $2.4\%$  and  $3.37\%$ ,  $0.92\%$  reaching  $82.13\%$ ,  $87.98\%$  and  $96.19\%$  accuracy in SST, IMDB and SVA, respectively. In fact, the results demonstrate consistent trends with our earlier MT experiments: additive window attention outdoes its multiplicative counterpart in the encoder,
As specified in https://github.com/tensorflow/tensor2tensor.
6596
Model	Perplexity
Vaswani et al. (2017)	46.37
Shaw et al. (2018)	46.13
Dec (MW)	44.00
Dec (AW)	44.95
Table 6: Perplexity scores on 1-billion-word language modeling benchmark (the lower the better).
where the entire key sequence is available.
5.4 Language Modeling
Finally, to demonstrate our proposed methods as effective general purpose NLP components, we evaluate them on the One Billion Word LM Benchmark dataset (Chelba et al., 2013). The dataset contains 768 million words of data compiled from WMT 2011 News Crawl data, with a vocabulary of 32,000 words. We used its held-out data as the test set.
Training Setup. As the LM dataset is considerably large, we used the same model settings as adopted in our MT experiments. For these experiments, we only trained the models on virtually 4 GPUs for 100,000 updates using gradient aggregation on a single GPU machine. Note that only the self-attention based autoregressive decoder of the Transformer framework is used in this task. Therefore, the method of Yang et al. (2018) is not applicable to this task.
Results. Table 6 shows the perplexity scores. As can be seen, our multiplicative and additive window attention models both surpass Transformer (Vaswani et al., 2017) by 2.37 and 1.42 points respectively, reaching 44.00 and 44.95 perplexity scores respectively. In addition, it is noteworthy that similar to MT experiments, multiplicative attention outperforms the additive one on this task, where the decoder is used. This further reinforces the claim that where the global context is not fully available like in the decoder, the incomplete global context may induce noises into the model. Thus, it is effective to embrace dynamic local window attention to suppress the global context, for which the multiplicative window attention is designed.
We have presented a novel Differential Window method for dynamic window selection, and used it
to improve the standard attention modules by enabling more focused attentions. Specifically, we proposed Trainable Soft Masking and Segment-based Masking, which can be applied to encoder/decoder self-attention and cross attention.
We evaluated our models on four NLP tasks including machine translation, sentiment analysis, subject verb agreement and language modeling. Our experiments show that our proposed methods outperform the baselines significantly across all the tasks. All in all, we demonstrate the benefit of incorporating the differentiable window in the attention. In the future, we would like to extend our work to make a syntactically-aware window that can automatically learn tree (or phrase) structures.
We would like to express our gratitude to Yi Tay and our anonymous reviewers for their insightful feedback on our paper. Shafiq Joty would like to thank the funding support from his Start-up Grant (M4082038.020).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR.
Yonatan Belinkov, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass. 2017. What do neural machine translation models learn about morphology? In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861-872, Vancouver, Canada. Association for Computational Linguistics.
Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Phillip Koehn, and Tony Robinson. 2013. One billion word benchmark for measuring progress in statistical language modeling. Technical report, Google.
Chung-Cheng Chiu and Colin Raffel. 2018. Monotonic chunkwise attention. In International Conference on Learning Representations.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure
6597
of language? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3651-3657, Florence, Italy. Association for Computational Linguistics.
Tal Linzen, Emmanuel Dupoux, and Yoav Goldberg. 2016. Assessing the ability of lstms to learn syntax-sensitive dependencies. Transactions of the Association for Computational Linguistics, 4:521-535.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), EMNLP, pages 1412-1421. ACL.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation (WMT).
Matthew Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. 2018. Dissecting contextual word embeddings: Architecture and representation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 1499-1509, Brussels, Belgium. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725. Association for Computational Linguistics.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on
Empirical Methods in Natural Language Processing, pages 1631-1642. Association for Computational Linguistics.
Matthias Sperber, Jan Niehues, Graham Neubig, Sebastian Stuker, and Alex Waibel. 2018. Self-attentional acoustic models. In *Interspeech* 2018, 19th Annual Conference of the International Speech Communication Association, Hyderabad, India, 2-6 September 2018., pages 3723-3727.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 2692-2700. Curran Associates, Inc.
Shuohang Wang and Jing Jiang. 2017. Machine comprehension using match-lstm and answer pointer. In 5th International Conference on Learning Representations, ICLR 2017, Toulouse, France, April 24-26, 2017, Conference Track Proceedings.
Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015a. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2048-2057, Lille, France. PMLR.
Kelvin Xu, Jimmy Lei Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhutdinov, Richard S. Zemel, and Yoshua Bengio. 2015b. Show, attend and tell: Neural image caption generation with visual attention. In Proceedings of the 32Nd International Conference on International Conference on Machine Learning - Volume 37, ICML'15, pages 2048-2057. JMLR.org.
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4449–4458, Brussels, Belgium. Association for Computational Linguistics.
6598
Proof:  $\hat{m}_q = \mathbb{E}(\pmb {m}_q)$
The probability of left and right boundary for a query  $q$ :
$$
\hat {\phi} _ {l _ {q}} = \mathcal {S} \left(\frac {\boldsymbol {q} ^ {T} \boldsymbol {W} _ {L} ^ {Q} \left(\boldsymbol {K} \boldsymbol {W} _ {L} ^ {K}\right) ^ {T}}{\sqrt {d}}\right) \tag {18}
$$
$$
\hat {\phi} _ {r _ {q}} = \mathcal {S} \left(\frac {\boldsymbol {q} ^ {T} \boldsymbol {W} _ {R} ^ {Q} \left(\boldsymbol {K} \boldsymbol {W} _ {R} ^ {K}\right) ^ {T}}{\sqrt {d}}\right) \tag {19}
$$
For any  $k$
$$
p \left(f _ {k} = 1\right) = p \left(l _ {q} \leq k\right) = \sum_ {\hat {\phi} _ {l _ {q}} \leq k} \hat {\phi} _ {l _ {q}} = \left(\hat {\phi} _ {l _ {q}} ^ {T} \boldsymbol {L} _ {n}\right) _ {k} (2 0)
$$
$$
p \left(g _ {k} = 1\right) = p \left(r _ {q} \geq k\right) = \sum_ {\hat {\phi} _ {r _ {q}} \geq k} \hat {\phi} _ {r _ {q}} = \left(\hat {\phi} _ {r _ {q}} ^ {T} \boldsymbol {L} _ {n} ^ {T}\right) _ {k} (2 1)
$$
Since  $f_{k}$  and  $g_{k}$  are binary values,
$$
\hat {f} _ {k} = p \left(f _ {k} = 1\right) = \mathbb {E} \left(f _ {k}\right) \tag {22}
$$
$$
\hat {g} _ {k} = p \left(g _ {k} = 1\right) = \mathbb {E} \left(g _ {k}\right) \tag {23}
$$
$$
\hat {\boldsymbol {m}} _ {q} = \hat {\boldsymbol {f}} _ {l _ {q}} \odot \hat {\boldsymbol {g}} _ {r _ {q}} + \hat {\boldsymbol {f}} _ {r _ {q}} \odot \hat {\boldsymbol {g}} _ {l _ {q}} = \mathbb {E} \left(\boldsymbol {m} _ {q}\right) \tag {24}
$$