SlowGuess/NAACL_2021_OCR huggingface dataset:
In this section, we conduct further analysis for our model. We first investigate stacking methods for different sublayers in § 4.1. Then we compare strategies of static mask and dynamic mask in § 4.2. Finally, we analyse the behavior of SAN and DMAN in localness modeling through attention scores in § 4.3.
4.1 Investigate Stacking Methods for Different Sublayers
Here, we investigate different collaboration mechanisms of the elements in MANs. Under our design principles, there are three elements: FFN, SAN, and DMAN. For the convenience of comparison, we take FFN as the last component in the sequential layered structure. We try different collaboration methods and test them on IWSLT2014 German-to-English (De-En). The results are shown in the Table 3. We conclude that:
1. Our proposed  $C\#5$  achieves the best performance that verifies the effectiveness of our proposed sequential layered structure.
2. All of  $C\# 3$ ,  $C\# 4$  and  $C\# 5$  outperform  $C\# 1$  and  $C\# 2$ , and the least improvement in BLEU is 0.2. This shows that no matter what collaboration method, models with the participation of DMAN perform better than models without DMAN, which validates the capability of DMAN.
3. Both  $C\# 5$  and  $C\# 4$  are better than  $C\# 3$  and  $C\# 2$ . This indicates that models without DMAN or SAN are not comparable to models with all three modules. This shows that DMAN and SAN have their own strengths, namely, localness modeling and globalness modeling, and are able to make up for each other's defects through collaboration.
4.  $C\# 5$  is better than  $C\# 4$ . This indicates that
'https://github.com/pltrdy/files2rouge
Model	CNN/Daily Mail	Gigaword
R-1	R-2	R-L	R-avg	R-1	R-2	R-L
LEAD-3 (Nallapati et al., 2016)	40.42	17.62	36.67	31.57	-	-	-
PTGEN+Coverage (See et al., 2017)	39.53	17.28	36.38	31.06	-	-	-
Dynamic Conv (Wu et al., 2019)	39.84	16.25	36.73	30.94	-	-	-
Transformer (Vaswani et al., 2017)	39.50	16.06	36.63	30.73	37.57	18.90	34.69
Ours	40.98	18.29	37.88	32.38	38.28	19.46	35.46
Table 2: Evaluation results on CNN/Daily Mail and Gigaword. R is short for ROUGE.
#	Method	BLEU
C#1	FFN→SAN→FFN	35.51
C#2	SAN→SAN→FFN	35.66
C#3	DMAN→DMAN→FFN	35.86
C#4	SAN→DMAN→FFN	35.91
C#5	DMAN→SAN→FFN	36.35
first modeling the localness and then globalness would be better than the inverse order.
4.2 Static Mask and Dynamic Mask
In this section, we compare the performance of Static Mask Attention Network (SMAN) and Dynamic Mask Attention Network (DMAN). Both of them follow the collaboration strategy of DMAN(SMAN)  $\rightarrow$  SAN  $\rightarrow$  FFN. In SMAN, we set a fixed mask boundary which has been determined in advance following Equation 11. Empirically, we propose two static mask strategies: (a)  $\mathrm{SMAN}_1$ , the boundary  $b$  depends on sentence length  $L$ ,  $b = \sqrt{L} /2$ ; (b)  $\mathrm{SMAN}_2$ ,  $b$  is set to 4, which is chosen from 2, 4, 6, 8 through validation.
The results in IWSLT2014 De-En are shown in Table 4. The performance of  $\mathrm{SMAN}_1$  and  $\mathrm{SMAN}_2$  are very close. They both outperform the Transformer but fall behind our proposed DMAN. This indicates that our proposed DMAN is superior to SMAN. SMAN fails to manage various neighborhood for different query tokens, but DMAN can model localness with more flexibility according to these factors.
Table 3: Performance of different collaboration methods of DMAN, SAN and FFN. We evaluate on IWSLT2014 De-En.
model	BLEU
Transformer	34.40
SMAN1	35.52
SMAN2	35.55
DMAN	36.35
Table 4: Performance of SMAN and DMAN on IWSLT2014 De-En.
4.3 Analysis of DMAN in Localness Modeling
In this section, we analyse the behavior of DMAN and SAN in localness modeling through attention scores in Equation 4. To quantify the role of neighbors in semantic modeling, we compute the sum of attention scores within some particular window size. Generally, if the attention score from  $a$  to  $c$  is bigger than  $b$  to  $c$ , we consider that  $a$  contributes more to the semantic modeling of  $c$  compared to  $b$ , in other words, model utilizes more information of  $a$  than  $b$  to learn the semantic representation of  $c$ . Therefore, larger attention scores mean that model utilizes more information of the corresponding tokens to learn the semantic representation of query token.
For each sentence in dataset  $X_{i} = (x_{i,1},\dots ,$ $x_{i,T_i})\in \mathcal{D}$  we utilize  $\bar{s}_{i,DMAN}^{l}$  and  $\bar{s}_{i,SAN}^{l}\in \mathbb{R}^{T_i\times T_i}$  to denote the average attention scores  $S_M(Q,K)$  in Equation 4 across different heads in the  $l$  -th layer for DMAN and SAN, respectively. We sum the attention scores of these tokens  $x_{i,k}$  within the window size  $w$  of the query  $x_{i,j}$  in the  $l$  -th layer, and average the sum across  $X_{i}$  and dataset  $\mathcal{D}$  following Equation 13.
$$
a t t n _ {-} s _ {w, l, *} = \frac {1}{| \mathcal {D} |} \sum_ {X _ {i} \in \mathcal {D}} \frac {1}{T _ {i}} \sum_ {x _ {i, j} \in X _ {i}} \sum_ {| k - j | \leq w} \bar {s} _ {i, *} ^ {l} [ j, k ] \tag {13}
$$
where  $*\in \{DMAN,SAN\}$ , and  $\bar{s}_{i,*}^l [j,k]$  is the
w	#1	#3	#6
DMAN	1	76.58	60.43	60.86
SAN	1	12.80	40.39	45.55
DMAN	2	86.17	75.56	73.89
SAN	2	18.73	45.62	52.72
DMAN	4	95.09	86.20	85.58
SAN	4	30.38	55.17	62.77
Table 5: The values of attention scores  $\text{attn\_s}_{w,l,\text{DMAN}}$  and  $\text{attn\_s}_{w,l,\text{SAN}}$ , which is shown in Equation 13.  $\mathcal{D}$  is the test set of IWSLT14 De-En, window size  $w = 1,2,4$  and encoder layers  $l = 1,3,6$ .
value of the  $j$ -th row and  $k$ -th column of  $\overline{s}_{i,*}^{l}$ .  $attn\_s_{w,l,*}$  measures the overall contribution of these neighbor tokens within the window size  $w$  to the query tokens' semantic modeling. We take  $\mathcal{D}$  as the test set of IWSLT14 De-En and compute  $attn\_s_{w,l,*}$  with  $w = 1,2,4$  and  $l = 1,3,6$ .
The result is shown in Table 5. We see that in layer#1, #3 and #6, the sum attention scores of DMAN within the window size 2 are  $50\%$  more than those of SAN, especially in layer#1 where the gap is as much as five times between SAN and DMAN. This phenomenon validates that the attention scores of DMAN in neighbors are larger than those of SAN, thus DMAN is more specialized in localness modeling than SAN.
Recently, there is a large body of work on improving Transformer (Vaswani et al., 2017) for various issues. For recurrence modeling, Hao et al. (2019) introduces a novel attentive recurrent network to leverage the strengths of both attention and recurrent networks. For context modeling, Yang et al. (2019a) focuses on improving self-attention through capturing the richness of context and proposes to contextualize the transformations of the query and key layers. Wu et al. (2019) introduces dynamic convolutions to predict separate convolution kernels solely based on the current time-step in order to determine the importance of context elements. In order to adjust attention weights beyond SAN, Shaw et al. (2018) extends the self-attention mechanism to efficiently consider representations of the relative positions or distances between sequence elements through adding a relative position embedding to the key vectors; Bugliarello and
Okazaki (2019) transfers the distance between two nodes in dependency trees with a pre-defined Gaussian weighting function and multiply the distance with the key-query inner product value; Dai et al. (2019) presents a relative position encoding scheme that adds additional relative position representation to the key-query computation. Sukhbaatar et al. (2019a) proposes a parameterized linear function over self-attention to learn the optimal attention span in order to extend significantly the maximum context size used in Transformer. To merge FFN to SAN, Sukhbaatar et al. (2019b) proposes a new model that solely consists of attention layers and augments the self-attention layer with persistent memory vectors that play a similar role as the feedforward layer. As for the collaboration of SAN and FFN, Lu et al. (2019) introduces Macaron layer that split the FFN into two half-steps based on Strang-Marchuk splitting scheme in ODE. For localness modeling, Yang et al. (2018) casts localness modeling as a learnable Gaussian bias according to relative distance to external energy in softmax function as a new self-attention network. Zhao et al. (2019) explores parallel multi-scale representation learning to capture both long-range and short-range language structures with combination of convolution and self-attention. In our work, DMAN, SAN and FFN are unified in Mask Attention Networks, where DMAN is a supplement of SAN and FFN that specializes in localness modeling. Moreover, we investigate different collaboration mechanisms.
In this paper, we introduce Mask Attention Networks and reformulate SAN and FFN to point out they are two special cases with static mask in MANs. We analyze the the deficiency of SAN and FFN in localness modeling. Dynamic Mask Attention Network is derived from MANs for better local structure modeling. Considering the different specialities of SAN, FFN, and DMAN, we investigate a sequential layered structure DMAN  $\rightarrow$  SAN  $\rightarrow$  FFN for their collaboration. Compared with original Transformer, our proposed model achieves better performance in neural machine translation and abstract summarization. For future work, we consider adding structure information or external knowledge, e.g., dependency tree, with mask matrices in MANs.
This work was supported by China National Key R&D Program (No.2018YFC0831105).
Karim Ahmed, Nitish Shirish Keskar, and Richard Socher. 2017. Weighted transformer network for machine translation. arXiv preprint arXiv:1711.02132.
Emanuele Bugliarello and Naoaki Okazaki. 2019. Improving neural machine translation with parent-scaled self-attention. arXiv preprint arXiv:1909.03149.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-XL: Attentive language models beyond a fixed-length context. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2978-2988, Florence, Italy. Association for Computational Linguistics.
Maosheng Guo, Yu Zhang, and Ting Liu. 2019. Gaussian transformer: a lightweight approach for natural language inference. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6489-6496.
Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. 2019. Modeling recurrence for transformer. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1198-1207, Minneapolis, Minnesota. Association for Computational Linguistics.
Chin-Yew Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using N-gram co-occurrence statistics. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 71-78. Association for Computational Linguistics.
Yiping Lu, Zhuohan Li, Di He, Zhiqing Sun, Bin Dong, Tao Qin, Liwei Wang, and Tie-Yan Liu. 2019. Understanding and improving transformer from a multi-particle dynamic system point of view. arXiv preprint arXiv:1906.02762.
Xutai Ma, Juan Miguel Pino, James Cross, Liezl Puzon, and Jiatao Gu. 2020. Monotonic multihead attention. In International Conference on Learning Representations.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th
SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations.
Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 379-389, Lisbon, Portugal. Association for Computational Linguistics.
Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and TieYan Liu. 2019. Mass: Masked sequence to sequence pre-training for language generation. In International Conference on Machine Learning, pages 5926-5936.
Sainbayar Sukhbaatar, Edouard Grave, Piotr Bojanowski, and Armand Joulin. 2019a. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 331-335, Florence, Italy. Association for Computational Linguistics.
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, and Armand Joulin. 2019b. Augmenting self-attention with persistent memory. arXiv preprint arXiv:1907.01470.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations.
Baosong Yang, Jian Li, Derek F Wong, Lidia S Chao, Xing Wang, and Zhaopeng Tu. 2019a. Context-aware self-attention networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 387-394.
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4449-4458, Brussels, Belgium. Association for Computational Linguistics.
Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019b. Convolutional self-attention networks. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4040–4045, Minneapolis, Minnesota. Association for Computational Linguistics.
Haoyu Zhang, Jingjing Cai, Jianjun Xu, and Ji Wang. 2019. Pretraining-based natural language generation for text summarization. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 789-797, Hong Kong, China. Association for Computational Linguistics.
Guangxiang Zhao, Xu Sun, Jingjing Xu, Zhiyuan Zhang, and Liangchen Luo. 2019. MUSE: Parallel multi-scale attention for sequence to sequence learning. arXiv preprint arXiv:1911.09483.