SlowGuess/NAACL_2021_OCR huggingface dataset:
Table 5: Results on the MIND dataset. *Improvement over the underlined second best results is significant at  $p < {0.05}$  .
of Transformer that uses direction- and distance-aware position encoding. The results of our approach and these methods on the five datasets are respectively shown in Tables 4 and 5. From the results, we have several observations.
First, compared with the vanilla Transformer, the compared methods that consider distance information consistently achieve better performance. It shows that distance information is very important in context modeling. Second, among the methods with distance information, the performance of Transformer-RPR is lower than the others. This may be because Transformer-RPR does not keep the precise long-distance information. Third, by comparing Transformer-XL and Adapted Transformer, we find that the performance of Adapted Transformer is better on the SST dataset, while Transformer-XL is better on other datasets. This is probably because Adapted Transformer is more suitable for modeling local contexts and the sentences in the SST dataset are usually short, while
Transformer-XL may be more appropriate for modeling long sequences. Fourth, our method consistently achieves better performance on the five datasets, and its improvement over the second best method is statistically significant (t-test  $p < 0.05$ ). This is because our method can explicitly encode real distance information rather than using positional encoding, making the modeling of distance more accurate.
We further compare the performance of different methods in a rating regression task on the Amazon dataset. The results are shown in Fig. 3. From Fig. 3 we observe similar patterns with the results in classification tasks, which validate the generality of our DA-Transformer in different genres of tasks.
4.3 Influence of Different Mapping Functions
Next, we study the influence of using different mapping functions  $f(\cdot)$  for computing the re-scaled coefficients. We compare the performance of our method w.r.t. several different  $f(\cdot)$ , including: (1)
Figure 3: Performance comparison of rating regression on Amazon. Lower scores indicate better performance.
$f(x) = \min (x,T)$  (clip), using a threshold  $T$  to clip the weighted distance; (2)  $f(x) = k_{i}x + b_{i}$  (linear), using a linear transformation to the weighted distance; (3)  $f(x) = \exp (x)$  (exponent), using an exponent function to map the weighted distance; (4)  $f(x) = \frac{1}{1 + \exp(-x)}$  (sigmoid), using the sigmoid function to activate the weighted distance; and (5)  $f(x;v_i) = \frac{1 + \exp(v_i)}{1 + \exp(v_i - x)}$ , our learnable sigmoid function. Due to space limitation, we only present the results on the AG, Amazon and MIND datasets in Fig. 4. From these results, we find that clip is not optimal for mapping the weighted distance. This is because it cannot keep the precise distance information beyond a certain range. In addition, simply using the linear transformation is also insufficient. This may be because our attention adjustment method requires  $f(\cdot)$  to be positive, but linear transformation cannot guarantee. Besides, we find that the sigmoid function and our proposed function are better than the exponential function. This may be because long sequences will lead to the problem of exponent explosion, which is harmful to context modeling. Moreover, our proposed learnable sigmoid function is better than the standard sigmoid function. It shows that adjusting the activation function in a learnable way can better map the raw distances into re-scaled coefficients.
4.4 Influence of Different Attention Adjusting Methods
Then, we explore the influence of different methods for adjusting the raw attention weights. We consider four different kinds of methods, including: (1) adding the re-scaled coefficients to the attention weights normalized by softmax (late add); (2) multiplying the re-scaled coefficients with the attention weights normalized by softmax (late multiply); (3) adding the re-scaled coefficients to the raw attention weights before normalization (early
add), which is widely used in existing methods like Transformer-XL; (4) multiplying the re-scaled coefficients with the raw attention weights activated by ReLU, which is the method used in our approach (early multiply). The results on the AG, Amazon and MIND datasets are shown in Fig. 5. According to these results, we find that early adjustment is better than late adjustment. This may be because the late adjustment methods will change the total amount of attention, which may not be optimal. In addition, we find that multiplying is better than adding for both early and late adjustment. This may be because adding large re-scaled coefficients may over-amplify some attention weights. For example, if a raw attention weight is relatively small, it is not suitable to add large re-scaled coefficients to it because the corresponding contexts may not have close relations. In contrast, multiplying the re-scaled coefficients will not over-amplify the low attention weights. Moreover, in our early multiply method we further propose to use the ReLU function to introduce sparsity to make the Transformer more "focused". Thus, our method is better than the existing early add method in adjusting the attention weights.
4.5 Model Interpretation
Finally, we interpret our proposed method by visualizing its key parameters and the attention weights. We first visualize the parameters  $w_{i}$  and  $v_{i}$  in our method, which control the preferences of attention heads on long-term or short-term information and the shape of the learnable sigmoid function, respectively. The visualization results on the AG and MIND datasets are respectively shown in Figs. 6 and 7. From Fig. 6, we find it is very interesting that half of the parameters  $w_{i}$  are positive and the rest of them are negative. It indicates that half of the attention heads mainly aim to capture local contexts, while the rest ones are responsible for modeling long-distance contexts. It may be because both short-term and long-term contexts are useful for understanding news topics. In addition, we find that most attention heads have negative  $v_{i}$  while the rest are positive. It shows that on the AG dataset the intensity of attention adjustment is mild in most attention heads. From Fig. 7(a), we find long-term information is somewhat more important than local information in modeling news texts for
<sup>6</sup>We show the average results of 5 runs. The values of  $w_{i}$  and  $v_{i}$  in these figures are sorted and are not corresponding to the head orders.
(a)  $AG$
(b) Amazon.
(c) MIND.
Figure 4: Influence of using different mapping functions.
(b) Amazon.
Figure 5: Influence of using different attention adjusting methods.
(a)  $w_{i}$
(b)  $v_{i}$
Figure 6: The weights learned by different attention heads on the AG dataset.
(a) Word-level  $w_{i}$ .
(b) News-level  $w_{i}$
(c) Word-level  $v_{i}$ .
Figure 7: The distance weights learned by different attention heads on the MIND dataset.
(d) News-level  $v_{i}$ .
news recommendation. However, from Fig. 7(b) we find an interesting phenomenon that only one
head has a strong negative  $w_{i}$  while the values of  $w_{i}$  in all the rest heads are positive. It means that only one attention head tends to capture short-term user interests while all the other heads prefer to capture long-term user interests. This is intuitive because users usually tend not to intensively click very similar news and their long-term interests may have more decisive influence on their news clicks. In addition, we find it is interesting that on MIND all values of  $v_{i}$  are positive. It may indicate that distance information has a strong impact on the attention weights. These visualization results show that DA-Transformer can flexibly adjust its preference on short-term or long-term information and the intensity of attention adjustment by learning different values of  $w_{i}$  and  $v_{i}$  according to the task characteristics. $^{7}$
We then visualize the attention weights produced by the vanilla Transformer and the distance-aware attention weights in our DA-Transformer method. The attention weights of a sentence in the  $AG$  dataset computed by four different attention heads are respectively shown in Figs. 8(a) and 8(b). From Fig. 8(a), we find it is difficult to interpret the self-attention weights because they are too "soft". In addition, it is difficult for us to understand the dif
<sup>7</sup>We do not observe significant correlations between the sequence length and the signs of  $w_{i}$ . This may indicate that the values of  $w_{i}$  depend more on the task characteristics rather than text lengths.
(a) Vanilla Transformer.
Figure 8: The self-attention weights learned by the vanilla Transformer and our proposed DA-Transformer method.
(b) DA-Transformer. The first two heatmaps are produced by heads with  $w_{i} < 0$  and others are produced by heads with  $w_{i} > 0$ .
ferences between the information captured by different attention heads. Different from the vanilla Transformer, from Fig. 8(b) we find that the attention weights obtained by our method are more sparse, indicating that the attention mechanism in our method is more focused. In addition, it is easier for us to interpret the results by observing the attention heatmap. For example, the first two heatmaps in Fig. 8(b) are produced by the two attention heads with preferences on short-term contexts. We can see that they mainly capture the relations among local contexts, such as the relations between "biotech" and "sector". Differently, in the latter two heatmaps obtained by the two attention heads that prefer long-term contexts, we can observe that the model tends to capture the relations between a word (e.g., "biotech") with the global contexts. These results show that different attention heads in our method are responsible for capturing different kinds of information, and their differences can be directly observed from the self-attention weights. Thus, our method can be better interpreted than vanilla Transformers.
In this paper, we propose a distance-aware Transformer, which can leverage the real distance between contexts to adjust the self-attention weights for better context modeling. We propose to first use different learnable parameters in different attention heads to weight the real relative distance between tokens. Then, we propose a learnable sigmoid function to map the weighted distances into re-scaled coefficients with proper ranges. They are further multiplied with the raw attention weights that are activated by the ReLU function to keep non-negativity and produce sharper attention. Extensive experiments on five benchmark datasets show that our approach can effectively improve the performance of Transformer by introducing real distance information to facilitate context modeling.
This work was supported by the National Natural Science Foundation of China under Grant numbers U1936208 and U1936216.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP, pages 632-642.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G Carbonell, Quoc Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. In ACL, pages 2978-2988.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*, pages 4171–4186.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep sparse rectifier neural networks. In AISTATS, pages 315-323.
Qipeng Guo, Xipeng Qiu, Pengfei Liu, Yunfan Shao, Xiangyang Xue, and Zheng Zhang. 2019. Start-transformer. In NAACL-HLT, pages 1315-1325.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR, pages 770-778.
Ruining He and Julian McAuley. 2016. Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering. In WWW, pages 507-517.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In *ICLR*.
Goro Kobayashi, Tatsuki Kuribayashi, Sho Yokoi, and Kentaro Inui. 2020. Attention is not only a weight: Analyzing transformers with vector norms. In EMNLP, pages 7057-7075.
Rik Koncel-Kedziorski, Dhanush Bekal, Yi Luan, Mirella Lapata, and Hannaneh Hajishirzi. 2019. Text generation from knowledge graphs with graph transformers. In NAACL-HLT, pages 2284-2293.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. Technical report, OpenAI.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In *NAACL-HLT*, pages 464–468.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, pages 1631-1642.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS, pages 5998-6008.
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. 2019. Self-attention with structural position representations. In EMNLP-IJCNLP, pages 1403-1409.
Chuhan Wu, Fangzhao Wu, Suyu Ge, Tao Qi, Yongfeng Huang, and Xing Xie. 2019. Neural news recommendation with multi-head self-attention. In EMNLP-IJCNLP, pages 6390-6395.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020a. Improving attention mechanism with query-value interaction. arXiv preprint arXiv:2010.03766.
Chuhan Wu, Fangzhao Wu, Tao Qi, and Yongfeng Huang. 2020b. Sentirec: Sentiment diversity-aware neural news recommendation. In AACL-IJCNLP, pages 44-53.
Fangzhao Wu, Ying Qiao, Jiun-Hung Chen, Chuhan Wu, Tao Qi, Jianxun Lian, Danyang Liu, Xing Xie, Jianfeng Gao, Winnie Wu, et al. 2020c. Mind: A large-scale dataset for news recommendation. In ACL, pages 3597-3606.
Hu Xu, Bing Liu, Lei Shu, and S Yu Philip. 2019. Bert post-training for review reading comprehension and aspect-based sentiment analysis. In *NAACL-HLT*, pages 2324-2335.
Hang Yan, Bocao Deng, Xiaonan Li, and Xipeng Qiu. 2019. Tener: Adapting transformer encoder for name entity recognition. arXiv preprint arXiv:1911.04474.
Baosong Yang, Jian Li, Derek F Wong, Lidia S Chao, Xing Wang, and Zhaopeng Tu. 2019. Context-aware self-attention networks. In AAAI, volume 33, pages 387-394.