SlowGuess/NAACL_2021_OCR huggingface dataset:
Chuhan Wu† Fangzhao Wu‡ Yongfeng Huang†
$^{\ddagger}$ Microsoft Research Asia, Beijing 100080, China
{wuchuhan15,wufangzhao}@gmail.com，yfhuang@tsinghua.edu.cn
Transformer has achieved great success in the NLP field by composing various advanced models like BERT and GPT. However, Transformer and its existing variants may not be optimal in capturing token distances because the position or distance embeddings used by these methods usually cannot keep the precise information of real distances, which may not be beneficial for modeling the orders and relations of contexts. In this paper, we propose DA-Transformer, which is a distance-aware Transformer that can exploit the real distance. We propose to incorporate the real distances between tokens to re-scale the raw self-attention weights, which are computed by the relevance between attention query and key. Concretely, in different self-attention heads the relative distance between each pair of tokens is weighted by different learnable parameters, which control the different preferences on long- or short-term information of these heads. Since the raw weighted real distances may not be optimal for adjusting self-attention weights, we propose a learnable sigmoid function to map them into re-scaled coefficients that have proper ranges. We first clip the raw self-attention weights via the ReLU function to keep non-negativity and introduce sparsity, and then multiply them with the rescaled coefficients to encode real distance information into self-attention. Extensive experiments on five benchmark datasets show that DA-Transformer can effectively improve the performance of many tasks and outperform the vanilla Transformer and its several variants.
Transformer (Vaswani et al., 2017) has achieved huge success in the NLP field in recent years (Kobayashi et al., 2020). It serves as the basic architecture of various state-of-the-art models like BERT (Devlin et al., 2019) and GPT (Radford et al., 2019), and boosts the performance of
many tasks like text generation (Koncel-Kedziorski et al., 2019), machine translation (Vaswani et al., 2017), and reading comprehension (Xu et al., 2019). Thus, the improvement on the Transformer architecture would be beneficial for many NLP-related fields (Wu et al., 2020a).
A core component of Transformer is multi-head self-attention, which is responsible for modeling the relations between contexts (Yang et al., 2019; Guo et al., 2019). However, self-attention is position-agnostic since it does not distinguish the orders of inputs. Thus, in the vanilla Transformer, position encoding is applied to the input to help Transformer capture position information. However, in contrast to recurrent and convolutional neural networks, it is difficult for vanilla Transformers to be aware of the token distances (Shaw et al., 2018), which are usually important cues for context modeling. Thus, several works explored to incorporate token distance information into Transformer. For example, Shaw et al. (2018) proposed to combine the embeddings of relative positions with attention key and value in the self-attention network. They restricted the maximum relative distance to only keep the precise relative position information within a certain distance. Yan et al. (2019) proposed a variant of self-attention network for named entity recognition, which incorporates sinusoidal embeddings of relative position to compute attention weights in a direction- and distance-aware way. However, the distance or relative position embeddings used by these methods usually cannot keep the precise information of the real distance, which may not be beneficial for the Transformer to capture word orders and the context relations.
In this paper, we propose a distance-aware Transformer (DA-Transformer), which can explicitly exploit real token distance information to enhance context modeling by leveraging the relative distances between different tokens to re-scale the raw attention weights before softmax normalization.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2059-2068 June 6-11, 2021. ©2021 Association for Computational Linguistics
More specifically, since global and local context modeling usually have different distance preferences, we propose to learn a different parameter in different attention heads to weight the token distances, which control the preferences of attention heads on long or short distances. In addition, since the weighted distances may not have been restricted to a proper range, we propose a learnable sigmoid function to map the weighted distances into rescaled coefficients. They are further multiplied with the raw attention weights that are clipped by the ReLU function for keeping the non-negativity and introducing sparsity. We conduct extensive experiments on five benchmark datasets for different tasks, and the results demonstrate that our approach can effectively enhance the performance of Transformer and outperform its several variants with distance modeling.
- We propose a distance-aware Transformer that uses the real token distances to keep precise distance information in adjusting attention weights for accurate context modeling.
- We propose to use different parameters to weight real distances in different attention heads to control their diverse preferences on short-term or long-term information.
- We propose a learnable sigmoid function to map the weighted distances into re-scaled coefficients with proper ranges for better adjusting the attention weights.
- We conduct extensive experiments on five benchmark datasets and the results validate the effectiveness of our proposed method.
To make this paper self-contained, we first briefly introduce the architecture of Transformer, which was initially introduced to the machine translation task (Vaswani et al., 2017). It has become an important basic neural architecture of various state-of-the-art NLP models like BERT (Devlin et al., 2019) and GPT (Radford et al., 2019). The core component of Transformer is multi-head self-attention. It has  $h$  attention heads, where the parameters in each head are independent. For the  $i$ -th attention head, it takes a matrix  $\mathbf{H}$  as the input. It first uses three
independent parameter matrices  $\mathbf{W}_Q^{(i)},\mathbf{W}_K^{(i)}$  ,and  $\mathbf{W}_V^{(i)}$  to respectively transform the input matrix  $H$  into the input query  $\mathbf{Q}^{(i)}$  ,key  $\mathbf{K}^{(i)}$  and value  $\mathbf{V}^{(i)}$  which is formulated as follows:
$$
\mathbf {Q} ^ {(i)}, \mathbf {K} ^ {(i)}, \mathbf {V} ^ {(i)} = \mathbf {H} \mathbf {W} _ {Q} ^ {(i)}, \mathbf {H} \mathbf {W} _ {K} ^ {(i)}, \mathbf {H} \mathbf {W} _ {V} ^ {(i)}. (1)
$$
Then, it uses a scaled dot-product attention head to process its query, key and value, which is formulated as follows:
$$
\operatorname {A t t e n t i o n} \left(\mathbf {Q} ^ {(i)}, \mathbf {K} ^ {(i)}, \mathbf {V} ^ {(i)}\right) = \operatorname {s o f t m a x} \left(\frac {\mathbf {Q} ^ {(i)} \mathbf {K} ^ {(i) \top}}{\sqrt {d}}\right) \mathbf {V} ^ {(i)}, \tag {2}
$$
where  $d$  is the dimension of the vectors in the query and key. The outputs of the  $h$  attention heads are concatenated together and the final output is a linear projection of the concatenated representations, which is formulated as follows:
$$
\begin{array}{r l} \text {M u l t i h e a d} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) & = \operatorname {C o n c a t} \left(\operatorname {h e a d} _ {1}, \dots , \operatorname {h e a d} _ {\mathrm {h}}\right) \mathbf {W} _ {O}, \\ \text {w h e r e} & \operatorname {h e a d} _ {\mathrm {i}} = \operatorname {A t t e n t i o n} \left(\mathbf {Q} ^ {(i)}, \mathbf {K} ^ {(i)}, \mathbf {V} ^ {(i)}\right), \end{array} \tag {3}
$$
where  $\mathbf{W}_O$  is an output projection matrix. In the standard Transformer, a position-wise feed-forward neural network is further applied to the output of multi-head self-attention network. Its function is formulated as follows:
$$
F F N (\mathbf {x}) = \max  (0, \mathbf {x} \mathbf {W} _ {1} + \mathbf {b} _ {1}) \mathbf {W} _ {2} + \mathbf {b} _ {2}, \tag {4}
$$
where  $\mathbf{W}_1, \mathbf{W}_2, \mathbf{b}_1, \mathbf{b}_2$  are kernel and bias parameters. Transformer also employs layer normalization (Ba et al., 2016) and residual connection (He et al., 2016) techniques after the multi-head self-attention and feed-forward neural networks, which are also kept in our method.
Since self-attention network does not distinguish the order and position of input tokens, Transformer adds the sinusoidal embeddings of positions to the input embeddings to capture position information. However, position embeddings may not be optimal for distance modeling in Transformer because distances cannot be precisely recovered from the dot-product between two position embeddings.
2.2 Distance-aware Transformer
Instead of directly using the sinusoidal position embedding (Vaswani et al., 2017) or the absolute position embedding (Devlin et al., 2019), several variants of the Transformer explore to use the relative positions to better model the distance between contexts (Shaw et al., 2018; Wang et al., 2019; Dai et al., 2019; Yan et al., 2019). For example, Shaw
et al. (2018) proposed to add the embeddings of relative positions to the attention key and value to capture the relative distance between two tokens. They only kept the precise distance within a certain range by using a threshold to clip the maximum distance to help generalize to long sequences. Dai et al. (2019) proposed Transformer-XL, which uses another form of relative positional encodings that integrate content-dependent positional scores and a global positional score into the attention weights. Yan et al. (2019) proposed direction-aware sinusoidal relative position embeddings and used them in a similar way with Transformer-XL. In addition, they proposed to use the un-scaled attention to better fit the NER task. However, relative position embeddings may not be optimal for modeling distance information because they usually cannot keep the precise information of real token distances. Different from these methods, we propose to directly re-scale the attention weights based on the mapped relative distances instead of using sinusoidal position embeddings, which can explicitly encode real distance information to achieve more accurate distance modeling.
3 DA-Transformer
In this section, we introduce our proposed distance-aware Transformer (DA-Transformer) approach, which can effectively exploit real token distance information to enhance context modeling. It uses a learnable parameter to weight the real distances between tokens in each attention head, and uses a learnable sigmoid function to map the weighted distances into re-scaled coefficients with proper ranges, which are further used to adjust the raw attention weights before softmax normalization. The details of DA-Transformer are introduced in the following sections.
3.1 Head-wise Distance Weighting
Similar with the standard Transformer, the input of our model is also a matrix that contains the representation of each token, which is denoted as  $\mathbf{H} = [\mathbf{h}_1,\mathbf{h}_2,\dots,\mathbf{h}_N]$ , where  $N$  is the length of the sequence. We denote the real relative distance between the  $i$ -th and  $j$ -th positions as  $R_{i,j}$ , which is computed by  $R_{i,j} = |i - j|$ . We can then obtain the relative distance matrix  $\mathbf{R} \in \mathbb{R}^{N \times N}$  that describes the relative distance between each pair of positions. In each attention head, we use a learnable parameter  $w_i$  to weight the relative distance
Figure 1: The curves of our learnable sigmoid function under different  $v_{i}$ .
by  $\mathbf{R}^{(i)} = w_{i}\mathbf{R}$ , which will be further used to adjust the self-attention weights. In our method, we stipulate that a more positive  $\mathbf{R}^{(i)}$  will amplify the attention weights more strongly while a more negative  $\mathbf{R}^{(i)}$  will diminish them more intensively. Thus, a positive  $w_{i}$  means that this attention head prefers to capture long-distance information, while a negative  $w_{i}$  means that it focuses more on local contexts. By learning different values of  $w_{i}$ , different attention heads may have different preferences on capturing either short-term or long-term contextual information with different intensity.
3.2 Weighted Distance Mapping
Since the raw weighted distances may not be in the proper range for adjusting the attention weights, we need to map them into the re-scaled coefficients via a function  $\hat{\mathbf{R}}^{(i)} = f(\mathbf{R}^{(i)})$  that is suitable for adjusting the self-attention weights. However, it is not a trivial task to design the function  $f(\cdot)$  because it needs to satisfy the following requirements: (1)  $f(0) = 1$ . We stipulate that zero distances do not influence the self-attention weights. (2) The value of  $f(\mathbf{R}^{(i)})$  should be zero when  $\mathbf{R}^{(i)} \to -\infty$ . This requirement is to guarantee that if an attention head prefers to capture local information ( $w_i < 0$ ), the long-distance information should be surpassed. $^1$  (3) The value of  $f(\mathbf{R}^{(i)})$  should be limited when  $\mathbf{R}^{(i)} \to +\infty$ . This requirement is to ensure that the model is able to process long sequences without over-emphasize distant contexts. (4) The scale of  $f(\cdot)$  needs to be tunable. This aims to help the model better adjust the intensity of distance information. (5) The function  $f(\cdot)$  needs to be mono
Although the raw negative attention weights may be raised to 0 by  $f(\cdot)$ , the model can still surpass these attention weights after softmax by increasing the scale of other attention weights.
tone. To satisfy the five requirements above, we propose a learnable sigmoid function to map the weighted relative distances  $\mathbf{R}^{(i)}$ , which is formulated as follows:
$$
f \left(\mathbf {R} ^ {(i)}; v _ {i}\right) = \frac {1 + \exp \left(v _ {i}\right)}{1 + \exp \left(v _ {i} - \mathbf {R} ^ {(i)}\right)}, \tag {5}
$$
where  $v_{i}$  is a learnable parameter in this head that controls the upperbound and ascending steepness of this function. The curves of our learnable sigmoid function under several different values of  $v_{i}$  are plotted in Fig. 1. We can see that the proposed function satisfies all the requirements above. In addition, from this figure we find that if  $v_{i}$  is larger, the upperbound of the curve is higher, which means that distance information is more intensive. When  $v_{i} = 0$ , it is in fact identical to the standard sigmoid function except for the scaling factor of 2. By mapping the weighted distances  $\mathbf{R}^{(i)}$  via the function  $f(\cdot)$ , we can obtain the final re-scaled coefficients  $\hat{\mathbf{R}}^{(i)}$  in a learnable way. Several illustrative examples of the re-scaled coefficients under  $w_{i} = \pm 1$  and  $v_{i} = \pm 1$  are respectively shown in Figs. 2(a)-2(d). We can see that if  $w_{i}$  is positive, long-distance contexts are preferred while short-term contexts are surpassed. The situation is reversed if  $w_{i}$  turns to negative. In addition, the coefficients in Fig. 2(c) have larger dynamic ranges than the coefficients in Fig. 2(a), indicating that long-distance information is more dominant in Fig. 2(c). Moreover, the coefficients in Fig. 2(d) are "sharper" than those in Fig. 2(b), which indicates that the model tends to capture shorter distances.
3.3 Attention Adjustment
Then, we use the re-scaled coefficients to adjust the raw attention weights that are computed by the dot-product between the query and key, i.e.,  $\frac{\mathbf{Q}^{(i)}\mathbf{K}^{(i)\top}}{\sqrt{d}}$ . Different from existing methods that add the query-key dot-product with position or distance representations, in our approach we propose to multiply the re-scaled coefficients with the query-key dot-product. This is because for the tokens whose relations are very weak, if their re-scaled coefficients are large, their final attention weights will be over-amplified if we simply add the re-scaled coefficients to their raw attention weights. This is not optimal for modeling contextual information because the attention weights of irrelevant contexts cannot be fully surpassed. However, there are also some problems if we directly multiply the
(a)  $w_{i} = 1, v_{i} = -1$ .
(b)  $w_{i} = -1,v_{i} = -1$
(c)  $w_{i} = 1,v_{i} = 1$
(d)  $w_{i} = -1, v_{i} = 1$ .
Figure 2: The re-scaled coefficient matrices under different values of  $w_{i}$  and  $v_{i}$ . Dark regions indicate that the corresponding attention weights are promoted.
re-scaled coefficients  $\hat{\mathbf{R}}^{(i)}$  and the raw attention weights  $\frac{\mathbf{Q}^{(i)}\mathbf{K}^{(i)\top}}{\sqrt{d}}$ . This is because the sign of attention weights  $\frac{\mathbf{Q}^{(i)}\mathbf{K}^{(i)\top}}{\sqrt{d}}$  is indefinite and the multiplied results cannot accurately reflect the influence of distance information. Thus, we propose to add a ReLU (Glorot et al., 2011) activation function to the raw attention weights to keep non-negativity. In this way, the final output  $\mathbf{O}^{(i)}$  of an attention head can be formulated as follows:
$$
\mathbf {O} ^ {(i)} = \operatorname {s o f t m a x} \left(\frac {\operatorname {R e L U} \left(\mathbf {Q} ^ {(i)} \mathbf {K} ^ {(i) \top}\right) * \hat {\mathbf {R}} ^ {(i)}}{\sqrt {d}}\right) \mathbf {V} ^ {(i)}, \tag {6}
$$
where  $*$  represents element-wise product. The ReLU function can also introduce sparsity to the self-attention because only the positive attention weights can be amplified by the re-scaled coefficients, which makes the attention weights in our method sharper. We concatenate the output from the  $h$  independent attention heads, and project it into a unified output. In addition, we keep the same layer normalization and residual connection strategy as the standard Transformer.
3.4 Computational Complexity Analysis
Compared with the standard Transformer, the major additional time cost is brought by computing the re-scaled coefficients  $\hat{\mathbf{R}}^{(\mathrm{i})}$  and using them to adjust the attention weights. The theoretical time complexity of the two operations in each head is  $O(N^2)$ , which is much smaller than the time complexity of computing the attention weights, i.e.,  $O(N^2\times d)$ . In addition, both Eq. (5) and Eq. (6) in our approach can be computed in a vectorized
manner. Thus, the additional time consumption of our method is very light. Besides, the increase of parameters is also minimal because we only introduce  $2h$  additional parameters, which are usually ignorable compared with the projection matrices like  $\mathbf{W}_Q^{(i)}$ . Thus, our approach inherits the efficiency of the Transformer architecture.
Our experiments are conducted on five benchmark datasets for different tasks. Four of them are benchmark NLP datasets. The first one is AG's  $\mathrm{News}^2$  (denoted as AG), which is a news topic classification dataset. The second one is Amazon Electronics (He and McAuley, 2016) (denoted as Amazon), which is a dataset for review rating prediction. The third one is Stanford Sentiment Treebank (Socher et al., 2013) (denoted as SST). We use the binary classification version of this dataset. The fourth one is Stanford Natural Language Inference (Bowman et al., 2015) (SNLI) dataset, which is a widely used natural language inference dataset. The detailed statistics of these datasets are summarized in Table 1. In addition, we also conduct experiments on a benchmark news recommendation dataset named MIND (Wu et al., 2020c), aiming to validate the effectiveness of our approach in both text and user modeling. It contains the news impression logs of 1 million users from Microsoft  $\mathrm{News}^3$  from October 12 to November 22, 2019. The training set contains the logs in the first five weeks except those on the last day which are used for validation. The rest logs are used for test. The key statistics of this dataset are summarized in Table 2.
Dataset	# Train	# Dev.	# Test	# Classes	Avg. len.
AG	108k	12k	7.6k	4	44
Amazon	40k	5k	5k	5	133
SST	8k	1k	2k	2	19
SNLI	55k	10k	10k	2	22
Table 1: Statistics of AG, Amazon, SST and SNLI datasets.
# Users	1,000,000	Avg. title len.	11.52
# News	161,013	# Click samples	5,597,979
# Impressions	500,000	# Non-click samples	136,162,621
Table 2: Statistics of the MIND dataset.
In our experiments, we use the 300-dimensional Glove (Pennington et al., 2014) embeddings for word embedding initialization. The number of attention head is 16, and the output dimension of each attention is 16. We use one Transformer layer in all experiments. On the AG, SST and SNLI datasets, we directly apply Transformer-based methods to the sentences. On the Amazon dataset, since reviews are usually long documents, we use Transformers in a hierarchical way by learning sentence representations from words via a word-level Transformer first and then learning document representations from sentences via a sentence-level Transformer. On the MIND dataset, following (Wu et al., 2019, 2020b) we also use a hierarchical model architecture that first learns representations of historical clicked news and candidate news from their titles with a word-level Transformer, then learns user representations from the representations of clicked news with a news-level Transformer, and final matches user and candidate news representations to compute click scores. We use the same model training strategy with negative sampling techniques as NRMS (Wu et al., 2019). On all datasets we use Adam (Kingma and Ba, 2015) as the optimization algorithm and the learning rate is 1e-3. On the AG, Amazon, SST and SNLI datasets, accuracy and macro-Fscore are used as the performance metric. On the MIND dataset, following (Wu et al., 2019) we use the average AUC, MRR, nDCG@5 and nDCG@10 scores of all sessions as the metrics. Each experiment is repeated 5 times independently and the average results with standard deviations are reported.
We compare our proposed DA-Transformer method with several baseline methods, including: (1) Transformer (Vaswani et al., 2017), the vanilla Transformer architecture, where sinusoidal positional embeddings are used. (2) Transformer-RPR (Shaw et al., 2018), a variant of Transformer with relative position representations. (3) Transformer-XL (Dai et al., 2019), a variant of Transformer that consists of a segment-level recurrence mechanism and a sinusoidal relative position encoding scheme. (4) Adapted Transformer (Yan et al., 2019), a variant
4We do not use contextualized embeddings generated by language models like BERT because we mainly focus on validating the effectiveness of our Transformer architecture.
Both the word-level and news-level Transformers contain one self-attention layer.
$^{2}$ https://www.di.unipi.it/en/
<sup>3</sup>https://www.msn.com/en-us
Methods	AG	Amazon
Accuracy	Macro-F	Accuracy	Macro-F
Transformer	93.01±0.13	93.00±0.13	65.15±0.40	42.14±0.41
Transformer-RPR	93.14±0.12	93.13±0.13	65.29±0.38	42.40±0.40
Transformer-XL	93.35±0.10	93.34±0.11	65.50±0.40	42.88±0.43
Adapted Transformer	93.28±0.13	93.27±0.14	65.47±0.39	42.69±0.42
*DA-Transformer	93.72±0.11	93.70±0.12	66.38±0.39	44.29±0.40
Table 3: Results on AG and Amazon. *Improvement over the underlined second best results is significant at  $p < 0.05$ .
Methods	SST	SNLI
Accuracy	Macro-F	Accuracy	Macro-F
Transformer	89.67±0.22	89.59±0.24	81.45±0.30	81.42±0.31
Transformer-RPR	89.94±0.19	89.90±0.20	82.20±0.31	82.18±0.31
Transformer-XL	90.06±0.20	90.02±0.21	83.19±0.29	83.15±0.30
Adapted Transformer	90.15±0.19	90.10±0.1	82.35±0.28	82.31±0.30
*DA-Transformer	90.49±0.17	90.43±0.19	84.18±0.27	84.16±0.29
Table 4: Results on SST and SNLI. *Improvement over the underlined second best results is significant at  $p < {0.05}$  .
Methods	AUC	MRR	nDCG@5	nDCG@10
Transformer	67.76±0.18	33.05±0.16	35.94±0.19	41.63±0.20
Transformer-RPR	67.81±0.16	33.10±0.17	35.98±0.20	41.65±0.21
Transformer-XL	67.92±0.16	33.15±0.16	36.04±0.20	41.70±0.19
Adapted Transformer	67.70±0.22	33.01±0.20	35.89±0.17	41.58±0.23
*DA-Transformer	68.32±0.15	33.36±0.16	36.34±0.14	42.07±0.17