SlowGuess/ACL_2020_OCR huggingface dataset:
Xinwei Geng $^{1*}$  Longyue Wang $^{2}$  Xing Wang $^{2}$  Bing Qin $^{1,3}$  Ting Liu $^{1,3}$  Zhaopeng Tu $^{2}$
$^{1}$ Harbin Institute of Technology  $^{2}$ Tencent AI Lab  $^{3}$ Peng Cheng Laboratory
$^{1}\{xwgeng, qinb, tliu\} @ir.hit.edu.cn$
$2\{v i n n y l y w a n g, b r i g h t x w a n g, z p t u \} @ t e n c e n t. c o m$
Self-attention networks (SANs) with selective mechanism has produced substantial improvements in various NLP tasks by concentrating on a subset of input words. However, the underlying reasons for their strong performance have not been well explained. In this paper, we bridge the gap by assessing the strengths of selective SANs (SSANs), which are implemented with a flexible and universal Gumbel-Softmax. Experimental results on several representative NLP tasks, including natural language inference, semantic role labelling, and machine translation, show that SSANs consistently outperform the standard SANs. Through well-designed probing experiments, we empirically validate that the improvement of SSANs can be attributed in part to mitigating two commonly-cited weaknesses of SANs: word order encoding and structure modeling. Specifically, the selective mechanism improves SANs by paying more attention to content words that contribute to the meaning of the sentence. The code and data are released at https://github.com/xwgeng/SSAN.
Self-attention networks (SANs) (Lin et al., 2017) have achieved promising progress in various natural language processing (NLP) tasks, including machine translation (Vaswani et al., 2017), natural language inference (Shen et al., 2018b), semantic role labeling (Tan et al., 2018; Strubell et al., 2018) and language representation (Devlin et al., 2019). The appealing strength of SANs derives from high parallelism as well as flexibility in modeling dependencies among all the input elements.
Recently, there has been a growing interest in integrating selective mechanism into SANs, which has produced substantial improvements in a variety
of NLP tasks. For example, some researchers incorporated a hard constraint into SANs to select a subset of input words, on top of which self-attention is conducted (Shen et al., 2018c; Hou et al., 2019; Yang et al., 2019b). Yang et al. (2018) and Guo et al. (2019) proposed a soft mechanism by imposing a learned Gaussian bias over the original attention distribution to enhance its ability of capturing local contexts. Shen et al. (2018c) incorporated reinforced sampling to dynamically choose a subset of input elements, which are fed to SANs.
Although the general idea of selective mechanism works well across NLP tasks, previous studies only validate their own implementations in a few tasks, either on only classification tasks (Shen et al., 2018c; Guo et al., 2019) or sequence generation tasks (Yang et al., 2018, 2019b). This poses a potential threat to the conclusive effectiveness of selective mechanism. In response to this problem, we adopt a flexible and universal implementation of selective mechanism using Gumbel-Softmax (Jang et al., 2017), called selective self-attention networks (i.e., SSANs). Experimental results on several representative types of NLP tasks, including natural language inference (i.e., classification), semantic role labeling (i.e., sequence labeling), and machine translation (i.e., sequence generation), demonstrate that SSANs consistently outperform the standard SANs (§3).
Despite demonstrating the effectiveness of SSANs, the underlying reasons for their strong performance have not been well explained, which poses great challenges for further refinement. In this study, we bridge this gap by assessing the strengths of selective mechanism on capturing essentially linguistic properties via well-designed experiments. The starting point for our approach is recent findings: the standard SANs suffer from two representation limitation on modeling word order encoding (Shaw et al., 2018; Yang et al., 2019a)
* Work done when interning at Tencent AI Lab.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2986-2995 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
and syntactic structure modeling (Tang et al., 2018; Hao et al., 2019a), which are essential for natural language understanding and generation. Experimental results on targeted linguistic evaluation lead to the following observations:
- SSANs can identify the improper word orders in both local ( $\S 4.1$ ) and global ( $\S 4.2$ ) ranges by learning to attend to the expected words.
- SSANs produce more syntactic representations (§5.1) with a better modeling of structure by selective attention (§5.2).
- The selective mechanism improves SANs by paying more attention to content words that possess semantic content and contribute to the meaning of the sentence (§5.3).
2.1 Self-Attention Networks
SANs (Lin et al., 2017), as a variant of attention model (Bahdanau et al., 2015; Luong et al., 2015), compute attention weights between each pair of elements in a single sequence. Given the input layer  $\mathbf{H} = \{\mathbf{h}_1, \dots, \mathbf{h}_N\} \in \mathbb{R}^{N \times d}$ , SANs first transform the layer  $\mathbf{H}$  into the queries  $\mathbf{Q} \in \mathbb{R}^{N \times d}$ , the keys  $\mathbf{K} \in \mathbb{R}^{N \times d}$ , and the values  $\mathbf{V} \in \mathbb{R}^{N \times d}$  with three separate weight matrices. The output layer  $\mathbf{O}$  is calculated as:
$$
\mathbf {O} = \operatorname {A T T} (\mathbf {Q}, \mathbf {K}) \mathbf {V} \tag {1}
$$
where the alternatives to  $\mathrm{ATT}(\cdot)$  can be additive attention (Bahdanau et al., 2015) or dot-product attention (Luong et al., 2015). Due to time and space efficiency, we used the dot-product attention in this study, which is computed as:
$$
\mathrm {A T T} (\mathbf {Q}, \mathbf {K}) = s o f t m a x (\frac {\mathbf {Q K} ^ {T}}{\sqrt {d}}) \qquad (2)
$$
where  $\sqrt{d}$  is the scaling factor with  $d$  being the dimensionality of layer states (Vaswani et al., 2017).
2.2 Weaknesses of Self-Attention Networks
Despite SANs have demonstrated its effectiveness on various NLP tasks, recent studies empirically revealed that SANs suffer from two representation limitations of modeling word order encoding (Yang et al., 2019a) and syntactic structure modeling (Tang et al., 2018). In this work, we concentrate on these two commonly-cited issues.
Word Order Encoding SANs merely rely on attention mechanism with neither recurrence nor convolution structures. In order to incorporate sequence order information, Vaswani et al. (2017) proposed to inject position information into the input word embedding with additional position embedding. Nevertheless, SANs are still weak at learning word order information (Yang et al., 2019a). Recent studies have shown that incorporating recurrence (Chen et al., 2018; Hao et al., 2019b,c), convolution (Song et al., 2018; Yang et al., 2019b), or advanced position encoding (Shaw et al., 2018; Wang et al., 2019a) into vanilla SANs can further boost their performance, confirming its shortcomings at modeling sequence order.
Structure Modeling Due to lack of supervision signals of learning structural information, recent studies pay widespread attention on incorporating syntactic structure into SANs. For instance, Strubell et al. (2018) utilized one attention head to learn to attend to syntactic parents of each word. Towards generating better sentence representations, several researchers propose phrase-level SANs by performing self-attention across words inside a ngram phrase or syntactic constituent (Wu et al., 2018; Hao et al., 2019a; Wang et al., 2019b). These studies show that the introduction of syntactic information can achieve further improvement over SANs, demonstrating its potential weakness on structure modeling.
2.3 Selective Self-Attention Networks
In this study, we implement the selective mechanism on SANs by introducing an additional selector, namely SSANs, as illustrated in Figure 1. The selector aims to select a subset of elements from the input sequence, on top of which the standard self-attention (Equation 1) is conducted. We implement the selector with Gumbel-Softmax, which has proven effective for computer vision tasks (Shen et al., 2018a; Yang et al., 2019c).
Selector Formally, we parameterize selection action  $a \in \{\text{SELECT}, \text{DISCARD}\}$  for each input element with an auxiliary policy network, where SELECT indicates that the element is selected for self-attention while DISCARD represents to abandon the element. The output action sequence  $\mathbf{A} \in \mathbb{R}^N$  is calculated as:
$$
\pi (\mathbf {A}) = \operatorname {s i g m o i d} \left(\mathbf {E} _ {s}\right) \tag {3}
$$
$$
\mathbf {E} _ {s} = \mathbf {Q} _ {s} \mathbf {K} _ {s} ^ {T} \tag {4}
$$
Figure 1: Illustration of SSANs that select a subset of input elements with an additional selector network, on top of which self-attention is conducted. In this example, the word "talk" performs attention operation over input sequence, where the words "Bush", "held" and "Sharon" are chosen as the truly-significant words.
where  $\mathbf{Q}_s\in \mathbb{R}^{N\times d}$  and  $\mathbf{K}_s\in \mathbb{R}^{N\times d}$  are transformed from the input layer  $\mathbf{H}$  with distinct weight matrices. We utilize sigmoid as activation function to calculate the distribution for choosing the action SELECT with the probability  $\pi$  or DISCARD with the probability  $1 - \pi$ .
Gumbel Relaxation There are two challenges for training the selector: (1) the ground-truth labels indicating which words should be selected are unavailable; and (2) the discrete variables in  $\mathbf{A}$  lead to a non-differentiable objective function. In response to this problem, Jang et al. (2017) proposed Gumbel-Softmax to give a continuous approximation to sampling from the categorical distribution. We adopt a similar approach by adding Gumbel noise (Gumbel, 1954) in the sigmoid function, which we refer as Gumbel-Sigmoid. Since sigmoid can be viewed as a special 2-class case ( $\mathbf{E}_s$  and  $\mathbf{0}$  in our case) of softmax, we derive the Gumbel-Sigmoid as:
$$
\begin{array}{l} G u m b e l - S i g m o i d \left(\mathbf {E} _ {s}\right) \\ = \operatorname {s i g m o i d} \left(\left(\mathbf {E} _ {s} + \mathbf {G} ^ {\prime} - \mathbf {G} ^ {\prime \prime}\right) / \tau\right) \tag {5} \\ = \frac {\exp \left(\left(\mathbf {E} _ {s} + \mathbf {G} ^ {\prime}\right) / \tau\right)}{\exp \left(\left(\mathbf {E} _ {s} + \mathbf {G} ^ {\prime}\right) / \tau\right) + \exp \left(\mathbf {G} ^ {\prime \prime} / \tau\right)} \\ \end{array}
$$
where  $\mathbf{G}^{\prime}$  and  $\mathbf{G}^{\prime \prime}$  are two independent Gumbel noises (Gumbel, 1954), and  $\tau \in (0,\infty)$  is a temperature parameter. As  $\tau$  diminishes to zero, a sample from the Gumbel-Sigmoid distribution becomes cold and resembles the one-hot samples. At training time, we can use Gumbel-Sigmoid to obtain
differentiable sample  $\mathbf{A}$  as Gumbel-Sigmoid  $(\mathbf{E}_s)$ . In inference, we choose the action with maximum probability as the final output.
3 NLP Benchmarks
To demonstrate the robustness and effectiveness of the SSANs, we evaluate it in three representative NLP tasks: language inference, semantic role labeling and machine translation. We used them as NLP benchmarks, which cover classification, sequence labeling and sequence generation categories. Specifically, the performances of semantic role labeling and language inference models heavily rely on structural information (Strubell et al., 2018), while machine translation models need to learn word order and syntactic structure (Chen et al., 2018; Hao et al., 2019c).
Natural Language Inference aims to classify semantic relationship between a pair of sentences, i.e., a premise and corresponding hypothesis. We conduct experiments on the Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015), which has three classes: Entailment, Contradiction and Neutral.
We followed Shen et al. (2018b) to use a token2token SAN layer followed by a source2token SAN layer to generate a compressed vector representation of input sentence. The selector is integrated into the token2token SAN layer. Taking the premise representation  $s^p$  and the hypothesis vector  $s^h$  as input, their semantic relationship is represented by the concatenation of  $s^p, s^h, s^p - s^h$  and  $s^p \cdot s^h$ , which is passed to a classification module to generate a categorical distribution over the three classes. We initialize the word embeddings with 300D GloVe 6B pre-trained vectors (Pennington et al., 2014), and the hidden size is set as 300.
Semantic Role Labeling is a shallow semantic parsing task, which aims to recognize the predicate-argument structure of a sentence, such as "who did what to whom", "when" and "where". Typically, it assigns labels to words that indicate their semantic role in the sentence. Our experiments are conducted on CoNLL2012 dataset provided by Pradhan et al. (2013).
We evaluated selective mechanism on top of DEEPATT $^{1}$  (Tan et al., 2018), which consists of
<https://github.com/XMUNLP/Tagger.>
stacked SAN layers and a following softmax layer. Following their configurations, we set the number of SAN layers as 10 with hidden size being 200, the number of attention heads as 8 and the dimension of word embeddings as 100. We use the GloVe embeddings (Pennington et al., 2014), which are pretrained on Wikipedia and Gigaword, to initialize our networks, but they are not fixed during training. We choose the better feed-forward networks (FFN) variants of DEEPATT as our standard settings.
Machine Translation is a conditional generation task, which aims to translate a sentence from a source language to its counterpart in a target language. We carry out experiments on several widely-used datasets, including small English  $\Rightarrow$  Japanese (En  $\Rightarrow$  Ja) and English  $\Rightarrow$  Romanian (En  $\Rightarrow$  Ro) corpora, as well as a relatively large English  $\Rightarrow$  German (En  $\Rightarrow$  De) corpus. For En  $\Rightarrow$  De and En  $\Rightarrow$  Ro, we respectively follow Li et al. (2018) and He et al. (2018) to prepare WMT2014 $^2$  and IWSLT2014 $^3$  corpora. For En  $\Rightarrow$  Ja, we use KFTT $^4$  dataset provided by Neubig (2011). All the data are tokenized and then segmented into subword symbols using BPE (Sennrich et al., 2016) with 32K operations.
We implemented the approach on top of advanced TRANSFORMER model (Vaswani et al., 2017). On the large-scale  $\mathrm{En}\Rightarrow \mathrm{De}$  dataset, we followed the base configurations to train the NMT model, which consists of 6 stacked encoder and decoder layers with the layer size being 512 and the number of attention heads being 8. On the small-scale  $\mathrm{En}\Rightarrow \mathrm{Ro}$  and  $\mathrm{En}\Rightarrow \mathrm{Ja}$  datasets, we followed He et al. (2018) to decrease the layer size to 256 and the number of attention heads to 4.
For all the tasks, we applied the selector to the first layer of encoder to better capture lexical and syntactic information, which is empirically validated by our further analyses in Section 4.
Table 1 shows the results on the three NLP benchmarks. Clearly, introducing selective mechanism significantly and consistently improves performances in all tasks, demonstrating the universality and effectiveness of the selective mechanism for SANs. Concretely, SSANs relatively improve prediction accuracy over SANs by  $+0.8\%$  and  $+0.5\%$
Task	Size	SANs	SSANs	△
Natural Language Inference (Accuracy)
SNLI	550K	85.60	86.30	+0.8%
Semantic Role Labeling (F1 score)
CoNLL	312K	82.48	82.88	+0.5%
Machine Translation (BLEU)
En⇒Ro	0.18M	23.22	23.91	+3.0%
En⇒Ja	0.44M	31.56	32.17	+1.9%
En⇒De	4.56M	27.60	28.50	+3.3%
Table 1: Results on the NLP benchmarks. "Size" indicates the number of training examples, and "△" denotes relative improvements over the vanilla SANs.
respectively on the NLI and SRL tasks, showing their superiority on structure modeling. Shen et al. (2018c) pointed that SSANs can better capture dependencies among semantically important words, and our results and further analyses (\$5) provide supports for this claim.
In the machine translation tasks, SSANs consistently outperform SANs across language pairs. Encouragingly, the improvement on translation performance can be maintained on the large-scale training data. The relative improvements on the  $\mathrm{En} \Rightarrow \mathrm{Ro}$ ,  $\mathrm{En} \Rightarrow \mathrm{Ja}$ , and  $\mathrm{En} \Rightarrow \mathrm{De}$  tasks are respectively  $+3.0\%$ ,  $+1.9\%$ , and  $+3.3\%$ . We attribute the improvement to the strengths of SSANs on word order encoding and structure modeling, which are empirically validated in Sections 4 and 5.
Shen et al. (2018c) implemented the selection mechanism with the REINFORCE algorithm. Jang et al. (2017) revealed that compared with Gumbel-Softmax (Maddison et al., 2014), REINFORCE (Williams, 1992) suffers from high variance, which consequently leads to slow converge. In our preliminary experiments, we also implemented REINFORCE-based SSANs, but it underperforms the Gumbel-Softmax approach on the benchmarking  $\mathrm{En} \Rightarrow \mathrm{De}$  translation task (BLEU: 27.90 vs. 28.50, not shown in the paper). The conclusion is consistent with Jang et al. (2017), and we thus use Gumbel-Softmax instead of REINFORCE in this study.
4 Evaluation of Word Order Encoding
In this section, we investigate the ability of SSANs of capturing both local and global word orders on the bigram order shift detection (§4.1) and word reordering detection (§4.2) tasks.
$^{2}$ http://www.statmt.org/wmt14.  
 $^{3}$ https://wit3.fbk.eu/mt.php?release=2014-01.  
 $^{4}$ http://www.phontron.com/kftt.
Model	Layer	Acc.	\( \bigtriangleup \)
SANs	-	52.23	-
SSANs	1	62.55	+19.8%
2	53.73	+2.9%
3	54.65	+4.6%
4	54.29	+3.9%
5	54.78	+4.9%
6	54.23	+3.8%
Table 2: Results on the local bigram order shift detection task when SSANs are applied into different layers.
4.1 Detection of Local Word Reordering
Task Description Conneau et al. (2018) propose a bigram order shift detection task to test whether an encoder is sensitive to local word orders. Given a monolingual corpus, a certain portion of sentences are randomly extracted to construct instances with illegal word order. Specially, given a sentence  $X = \{x_{1},\ldots ,x_{N}\}$ , two adjacent words (i.e.,  $x_{n},x_{n + 1}$ ) are swapped to generate an illegal instance  $X^{\prime}$  as a substitute for  $X$ . Given processed data which consists of intact and inverted sentences, examined models are required to distinguish intact sentences from inverted ones. To detect the shift of bigram word order, the models should learn to recognize normal and abnormal word orders.
The model consists of 6-layer SANs and 3-layer MLP classifier. The layer size is 128, and the filter size is 512. We trained the model on the open-source dataset provided by Conneau et al. (2018). The accuracy of SAN-based encoder is higher than previously reported result on the same task (Li et al., 2019) (52.23 vs. 49.30).
Detection Accuracy Table 2 lists the results on the local bigram order shift detection task, in which SSANs are applied to different encoder layers. Clearly, all the SSANs variants consistently outperform SANs, demonstrating the superiority of SSANs on capturing local order information. Applying the selective mechanism to the first layer achieves the best performance, which improves the prediction accuracy by  $+19.8\%$  over SANs. The performance gap between the SSANs variants is very large (i.e.,  $19.8\%$  vs. around  $4\%$ ), which we attribute to that the detection of local word reorder depends more on lexical information embedded in the bottom layer.
Figure 2: Attention weights over attended words with different relative distance from the query word on the local reordering task. SSANs pay more attention to the adjacent words (distance = 1) than SANs.
(a) SANs
Figure 3: Visualization of attention weights from an example on the local reordering detection task. We highlight the attended word (Y-axis) with maximum attention weight for each query (X-axis) in red rectangles.
(b) SSANs
Attention Behaviors The objective of local reordering task is to distinguish the swap of two adjacent words, which requires the examined model to pay more attention to the adjacent words. Starting from this intuition, we investigate the attention distribution over the attended words with different relative distances from the query word, as illustrated in Figure 2. We find that both SANs and SSANs focus on neighbouring words (e.g., distance  $< 3$ ), and SSANs pays more attention to the adjacent words (distance  $= 1$ ) than SANs (14.6% vs. 12.4%). The results confirm our hypothesis that the selective mechanism helps to exploit more bigram patterns to accomplish the task objective. Figure 3 shows an example, in which SSANs attend most to the adjacent words except the inverted bigram "he what". In addition, the surrounding words "exactly" and "wanted" also pay more attention to the exceptional word "he". We believe such features help to distinguish the abnormally local word order.
$^{5}$ https://github.com/facebookresearch/ SentEval/tree/master/data/probing.
Model	Layer	Insert	Original	Both
SANs	-	73.20	66.00	60.10
SSANs	1	81.52	72.19	66.77
2	80.14	70.01	63.97
3	79.82	69.69	63.93
4	79.08	70.22	63.67
5	80.19	69.84	64.12
6	80.27	69.50	63.73
Table 3: Performance on the global word reordering detection (WRD) task.
4.2 Detection of Global Word Reordering
Task Description Yang et al. (2019a) propose a word reordering detection task to investigate the ability of SAN-based encoder to extract global word order information. Given a sentence  $X = \{x_{1},\ldots ,x_{N}\}$ , a random word  $x_{i}$  is popped and inserted into another position  $j$  ( $i \neq j$ ). The objective is to detect both the original position the word is popped out (labeled as "O"), and the position the word is inserted (labeled as "T").
The model consists of 6-layer SANs and a output layer. The layer size is 512, and the filter size is 2048. We trained the model on the open-source dataset provided by Yang et al. (2019a).
Detection Accuracy Table 3 lists the results on the global reordering detection task, in which all the SSANs variants improve prediction accuracy. Similarly, applying the selective mechanism to the first layer achieves the best performance, which is consistent with the results on the global word reordering task (Table 2). However, the performance gap between the SSANs variants is much lower that that on the local reordering task (i.e.,  $4\%$  vs.  $15\%$ ). One possible reason is that the detection of global word reordering may also need syntactic and semantic information, which are generally embedded in the high-level layers (Peters et al., 2018).
Attention Behaviors The objective of the WRD is to distinguish a global reordering (averaged distance is 8.7 words), which requires the examined model to pay more attention to distant words. Figure 4 depicts the attention distribution according to different relative distances. SSANs alleviate the leaning-to-local nature of SANs and pay more attention to distant words (e.g., distance  $>5$ ), which better accomplish the task of detecting global reordering. Figure 5 illustrates an example, in which
Figure 4: Attention weights over attended words with different relative distance from the query word on the global WRD task. SSANs pay more attention to the distant words (distance  $>5$ ) than SANs.
(a) SANs
Figure 5: Visualization of attention weights from an example on the global reordering detection task. We highlight the attended word (Y-axis) with maximum attention weight for each query (X-axis) in red rectangles.
more queries in SSANs attend most to the inserted word "the" than SANs. Particularly, SANs pay more attention to the surrounding words (e.g., distance  $< 3$ ), while the inserted word "the" only accepts subtle attention. In contrast, SSANs dispense much attention over words centred on the inserted position (i.e., "the") regardless of distance, especially for the queries "current rules for now". We speculate that SSANs benefits from such features on detecting the global word reordering.