SlowGuess/ACL_2020_OCR huggingface dataset:
Yekun Chai† Shuo Jin‡ Xinwen Hou†
†Institute of Automation, Chinese Academy of Sciences
$^{\ddagger}$ University of Pittsburgh
chaiyekun@gmail.com shj42@pitt.edu
Self-attention mechanisms have made striking state-of-the-art (SOTA) progress in various sequence learning tasks, standing on the multi-headed dot product attention by attending to all the global contexts at different locations. Through a pseudo information highway, we introduce a gated component self-dependency units (SDU) that incorporates LSTM-styled gating units to replenish internal semantic importance within the multi-dimensional latent space of individual representations. The subsidiary content-based SDU gates allow for the information flow of modulated latent embeddings through skipped connections, leading to a clear margin of convergence speed with gradient descent algorithms. We may unveil the role of gating mechanism to aid in the context-based Transformer modules, with hypothesizing that SDU gates, especially on shallow layers, could push it faster to step towards suboptimal points during the optimization process.
Self-attention mechanism has lately attracted extensive interests due to its remarkable achievement on a wide range of sequence modeling applications, including natural language processing such as neural machine translation (Vaswani et al., 2017; Ott et al., 2018; Shaw et al., 2018), language modeling (LM) (Dai et al., 2019; Al-Rfou et al., 2019), self-supervised pretraining (Radford et al., 2018; Devlin et al., 2018; Lan et al., 2019); image generation (Parmar et al., 2018); deep reinforcement learning (Zambaldi et al., 2018; Vinyals et al., 2019), etc.
Holding the great promise of deep neural networks in language and images, Transformer capitalizes on the stacked multi-headed self-attention mechanism based on the conventional encoder-decoder architecture in a sequence-to-sequence
(seq2seq) manner to learn the global soft signals without explicit recurrence mechanism. Multi-head dot product attention (MHDPA) not only underpins the parallel training of multiple heads but captures long-term dependencies across an arbitrarily long distance within the same context. In which separated multiple heads independently draw sub-level attentions within the latent semantic sub-space of a fixed dimension, where different heads are presumed to signal different meaning aspects implicitly (Vaswani et al., 2017). Additionally, residual connections between layers allow the deep tandem stack of multiple identical modules by impeding degradation problem during training (He et al., 2016). Thus Transformer architectures take the place of Recurrent Neural Networks (RNNs), especially Long Short-Term Memory (LSTM) networks (Hochreiter and Schmidhuber, 1997) to be the model solution to learning sequential data.
Recently, there have been plenty of works contending that gating mechanisms could play a vital role or even entirely substitute RNNs or Transformers to model language sequences. Dauphin et al. (2017) firstly claimed that non-recurrent networks are also highly competitive with conventional RNN-dominated models in LM. They proposed the hierarchical gated temporal convolution neural networks (CNNs) with Gated Linear Units (GLU) to replace the recurrent connections in RNNs and achieved strong performance with faster training speed. Gehring et al. (2017) integrated absolute positional embedding, multi-step attention, GLU, and residual connections into entirely convolutional models to outperform strong LSTM models in NMT and abstractive summarization tasks. Wu et al. (2019) applied dynamic convolutions using shared softmax-normalized filters of depth-wise on GLU-regulated inputs within a fixed reception field rather than global contexts, challenging the common self-attention-dominated intuition.
6887
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6887-6900 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
However, all of the models, as mentioned earlier, adopt stacked CNNs rather than self-attention networks (SAN) to attend to the global contexts. It is well-known that CNNs are good at learning local-region features rather than long-term dependency, while SANs are adept in attending global dependencies. Context-based self-attention can capture the importance of relative relations under a valid context and is thus location-unaware. It focuses on the object-wise attention distributions between any two words but ignores the fundamental importance of feature-wise information.
Intuitionally, people need to consider not only the global contextual dependency but the meaning of individual words to comprehend the reading materials better. Grounding on this, we apply self-gating approaches on Transformer blocks for seq2seq modeling that combines gating units with skip-connections and Transformers to jointly take into account both the inner feature-wise importance and the relation-aware content-based attention distribution.
We adopt the self-dependency gating approach to intrinsically draw a binary importance ratio of itself and decide how much information of each feature to retain or remove. Our key contributions are:
- to illustrate that our self-dependency units on shallow Transformer layers could expedite the convergence speed during both the training and validation process without hyperparameter tuning.
- to support the claim that Transformer layers in different depth attend to information of different aspects, wherein bottom layers focus on local-range encodings. It substantiates the argument that the bottom layers of SAN can learn more in local contexts (Yang et al., 2018).
- to empirically prove that self-gating mechanisms are complementary to recurrence mechanisms in R-Transformer and Transformer-XL components.
This section briefly introduces the related background of Transformer and Highway Networks.
SAN has been dominant in most SOTA sequence learning models, whose basic components consist of stacked Transformers modules. We conduct comparison experiments on the Transformer and
its two variants, Transformer-XL (Dai et al., 2019) and R-Transformer (Wang et al., 2019).
2.1 Multi-head Dot Product Attention
Scaled dot product attention (DPA) (Vaswani et al., 2017) computes global attention weights between pairs within the context across an arbitrarily long distance, which could allow the simultaneous training and space-saving, impeding the drawbacks of sequential dependency of RNNs.
Given the input word representation  $\mathbf{X} \in \mathbb{R}^{L \times dh}$ , where  $L$  is the sequence length,  $d$  is the input dimension of each head and  $h$  is the number of attention heads, DPA uses the linear projection to acquire the query  $\mathbf{Q}$ , key  $\mathbf{K}$  and value  $\mathbf{V}$ . Denoting splitted inputs for  $i$ -th head as  $\mathbf{X}_i \in \mathbb{R}^{L \times d}$ , where  $i \in \{1, \dots, h\}$ , single-head self-attention can be calculated as:
$$
\mathbf {Q} _ {i}, \mathbf {K} _ {i}, \mathbf {V} _ {i} = \mathbf {X} _ {i} \mathbf {W} _ {q}, \mathbf {X} _ {i} \mathbf {W} _ {k}, \mathbf {X} _ {i} \mathbf {W} _ {v} \tag {1}
$$
$$
\operatorname {h e a d} _ {i} = \operatorname {s o f t m a x} \left(d ^ {- 1 / 2} \mathbf {Q} _ {i} \mathbf {K} _ {i} ^ {\top}\right) \mathbf {V} _ {i} \tag {2}
$$
where learnable weights  $\{\mathbf{W}_q,\mathbf{W}_k,\mathbf{W}_v\} \in \mathbb{R}^{d\times d}$ $d^{-1 / 2}$  is a scaling factor to prevent the effect of large values. In LM tasks, attention weights before softmax function are masked to only attend to history sequences.
MHDPA (Fig 1a) linearly projects the single DPA into  $h$  heads and performs attention operation in parallel, to jointly learn different semantic meanings of different subspaces (Vaswani et al., 2017). MHDPA can be calculated as:
$$
\operatorname {A t t} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = \left[ \operatorname {h e a d} _ {1} \circ \dots \circ \operatorname {h e a d} _ {h} \right] \mathbf {W} _ {\mathbf {o}} \tag {3}
$$
where  $\circ$  denotes the concatenation of  $h$  different heads,  $\mathbf{W}_{\mathbf{o}} \in \mathbb{R}^{dh \times dh}$  is the trainable weight.
Absolute Positional Encoding Transformer applies sinusoidal timing signal as the absolute positional encoding (PE) and directly element-wise add the dense word embeddings  $\mathbf{E} \in \mathbb{R}^{L \times dh}$  on the PE before feeding into Transformer modules:
$$
P E _ {(\text {p o s}, 2 i)} = \sin \left(\frac {\text {p o s}}{1 0 0 0 0 ^ {2 i / d}}\right) \tag {4}
$$
$$
P E _ {(\text {p o s}, 2 i + 1)} = \cos \left(\frac {\text {p o s}}{1 0 0 0 0 ^ {2 i / d}}\right) \tag {5}
$$
$$
\mathbf {X} = \mathbf {E} + P E (\mathbf {E}) \tag {6}
$$
where 'pos' indicates the position of sequences,  $i$  denotes the order along the embedding dimension.
6888
Given input representations  $\mathbf{X}$ , Transformer components with a sternward Layer Normalization (LN) is:
$$
\mathbf {U} = L N (\mathbf {X} + \operatorname {A t t} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {7}
$$
$$
\operatorname {F F N} (\mathbf {U}) = F F \left(\operatorname {R e L U} (F F (\mathbf {U}))\right) \tag {8}
$$
$$
\mathbf {O} = L N (\mathbf {U} + \operatorname {F F N} (\mathbf {U})) \tag {9}
$$
where Eq. 8 indicates the position-wise feedforward networks (FFN),  $\mathbf{O} \in \mathbb{R}^{L \times dh}$  represents the output of transformer layer.  $FF$  denotes the feed-forward fully-connected layer, ReLU is used as the non-linear activate function.
2.3 Transformer-XL
Transformer-XL (Dai et al., 2019) injected relative PE and segment-level recurrence to provide historical information for LM tasks.
Relative Positional Encoding Transformer-XL decomposed the dot product calculation of MHDPA, merged terms with similar meanings of positional bias, and reduced trainable weights with global positional semantics. It incorporated partial trainable parameters of relative sinusoidal PE in the MHDPA operation.
The Relative PE  $A^{\mathrm{rel}}$  of Transformer-XL is:
$$
a = \mathbf {Q} ^ {\top} \mathbf {K} \tag {10}
$$
$$
b = \mathbf {Q} ^ {\top} \mathbf {W} _ {\mathbf {k}, \mathbf {R}} \mathbf {R} \tag {11}
$$
$$
c = \mathbf {u} ^ {\top} \mathbf {K} \tag {12}
$$
$$
d = \mathbf {v} ^ {\top} \mathbf {W} _ {\mathbf {k}, \mathbf {R}} \mathbf {R} \tag {13}
$$
$$
A ^ {\mathrm {r e l}} (\mathbf {Q}, \mathbf {K}) = a + b + c + d \tag {14}
$$
where  $\mathbf{W}_{\mathbf{k},\mathbf{R}}\in \mathbb{R}^{d\times d}$ ,  $\{\mathbf{u},\mathbf{v}\} \in \mathbb{R}^d$  are trainable parameters. For each two positions  $i,j$  in the segment,  $\mathbf{R}$  is sinusoidal encoding matrices between relative position  $i - j$ . The terms  $a,b,c,d$  in the Eq. 10, 11, 12, 13 represent the content-based addressing, content-dependent positional biases, global biases between different positions and the global positional biases, respectively.
Segment-level Recurrence In Transformer-XL, the previous hidden states are cached and reused to inject the history information and attend to contexts beyond a fixed length through multi-layer stacks.
The MHDPA is computed as:
$$
\mathbf {M} _ {\tau} ^ {n - 1} = \overbrace {S G \left(\mathbf {X} _ {\tau - 1} ^ {\mathbf {n} - \mathbf {1}}\right)} ^ {\text {s t o p g r a d i e n t}} \circ \mathbf {X} _ {\tau} ^ {\mathbf {n} - \mathbf {1}} \tag {15}
$$
$$
\mathbf {Q}, \mathbf {K}, \mathbf {V} = \mathbf {X} _ {\tau} ^ {n - 1} \mathbf {W} _ {\mathbf {q}}, \mathbf {M} _ {\tau} ^ {n - 1} \mathbf {W} _ {\mathbf {k}}, \mathbf {M} _ {\tau} ^ {n - 1} \mathbf {W} _ {\mathbf {v}} \tag {16}
$$
$$
D P A (\mathbf {Q}, \mathbf {K}, \mathbf {V}) = A ^ {\mathrm {r e l}} (\mathbf {Q}, \mathbf {K}) \mathbf {V} \tag {17}
$$
wherein the key and value  $\mathbf{M}_{\tau}^{n - 1}$  concatenate the previous memory  $\mathbf{X}_{\tau -1}^{\mathbf{n} - \mathbf{1}}$  with the current segment inputs  $\mathbf{X}_{\tau}^{\mathbf{n} - \mathbf{1}}$  for the  $\tau$  -th segment in the  $n$  -th layer,  $SG$  means no backpropagation through the tensor.
2.4 R-Transformer
R-Transformer (Wang et al., 2019) employed short-range RNNs, termed localRNNs, to capture the positional information without explicit PEs. localRNNs take the recurrent connections within a local context, and shift right with one position at each time step. It can be seen as applying the RNN cells, such as LSTM, on the same receptive fields as the convolutional filters along the sequence direction.
$$
\mathbf {X} = \operatorname {l o c a l R N N} (\mathbf {E}) \tag {18}
$$
$$
\mathbf {O} = \text {T r a n s f o r m e r - l a y e r} (\mathbf {X}) \tag {19}
$$
None of the above Transformer models explicitly consider the essential feature-wise information. We augment several gated units on the Transformer block of the models above and empirically illustrate the effectiveness of gating units on convergence acceleration.
2.5 Highway Networks
Let we define the non-linear transforms as  $H, T$  and  $C$ , Highway Network (Srivastava et al., 2015) is defined as:
$$
\mathbf {O} = H (\mathbf {X}) \odot T (\mathbf {X}) + \mathbf {X} \odot C (\mathbf {X}) \tag {20}
$$
where  $T(\cdot)$  and  $C(\cdot)$  denote transform and carry gates to control the input transformation,  $\odot$  denotes the Hadamard product.
3 Gating Architecture
LSTM-styled gate units have been proven to be effective on sequence learning tasks (Dauphin et al., 2017; Gehring et al., 2017; Wu et al., 2019). We spontaneously wonder whether such gating mechanisms could help when augmenting the Transformer components.
6889
(a) Multi-head dot product self-attention
(b) Self-dependency units
(c) SDU-augmented Transformer block
Figure 1: Illustration of MHDPA, SDU and SDU-enhanced Transformer block.
3.1 Self-Dependency Units
Similar to GLU (Dauphin et al., 2017) that adopts the inputs as sigmoidal gates, we apply the Self-Dependency Units (SDU) by taking full inputs as their respective self gates and computing the element-wise product upon themselves (Fig 1b).
$$
T (\mathbf {X}) = \boldsymbol {\Psi} \left(\mathbf {X} \mathbf {W} _ {1} + \mathbf {b} _ {1}\right) \tag {21}
$$
$$
\operatorname {S D U} (\mathbf {X}) = T (\mathbf {X}) \odot \left(\mathbf {X W} _ {2} + \mathbf {b} _ {2}\right) \tag {22}
$$
where  $T(\mathbf{X})$  indicates the transform gate,  $\Psi$  is the gate function that confine the linear projection into a fixed range,  $\{\mathbf{W}_1,\mathbf{W}_2\} \in \mathbb{R}^{d\times d}$  and  $\{\mathbf{b_1},\mathbf{b_2}\} \in \mathbb{R}^d$  are trainable parameters.
The element-wise gating function  $\Psi$  takes sigmoidal-curve functions to regulate the pointwise weights within a fixed region, which have a side effect of relative normalization. Specifically, the sigmoid function  $\sigma (x) = 1 / (1 + \exp (-x))$  and its rescaled version  $\tanh (x) = 2\sigma (2x) - 1$ , where  $x\in \mathbb{R}$ .
We interpret the tanh function as an update gate, which can restrict the importance range into between -1 and 1, while the  $\sigma$  function bears a resemblance to the input gate in LSTMs to modulate how much information to retain at the feature-wise level.
3.2 Pseudo-highway Connection
MHDPA computes the multi-headed pairwise attention along the sequence dimension by measuring the distance between each word. It might overlook the fundamental importance of individual features. Rather than replacing MHDPA as gating and convolution operations in dynamic convolutions (Wu et al., 2019), we simply add a new branch of inputs to enrich the representations of residual connected
MHDPA with augmented gating-modified encodings. The gated units are also supplemented on FFN modules to provide additional self-adaptive information flow (Fig 1c).
From other perspectives, SDU can be considered as a self-dependency non-linear activation function with dynamic adaptation. The self-gating augmented Transformer module is calculated as:
$$
\begin{array}{l} \mathbf {U} = L N \left(\mathbf {X} + \operatorname {A t t} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \right. \tag {23} \\ + \operatorname {S D U} (\mathbf {X})) \\ \end{array}
$$
$$
\mathbf {O} = L N \left(\mathbf {U} + \operatorname {F F N} (\mathbf {U}) + \operatorname {S D U} (\mathbf {U})\right) \tag {24}
$$
where  $\mathbf{U}$  and  $\mathbf{O}$  represent the intermediate representation and outputs.
Pseudo-highway Transformer When we take  $\sigma$  gate as  $\Psi$ , we can have the similar format as highway networks:
$$
\begin{array}{l} \nabla [ \mathbf {f} (\mathbf {X}) \odot \sigma (\mathbf {g} (\mathbf {X})) ] = \overbrace {\sigma (\mathbf {g} (\mathbf {X}))} ^ {\text {t r a n s f o r m g a t e}} \odot \nabla \mathbf {f} (\mathbf {X}) \\ + \overbrace {\left(1 - \sigma (\mathbf {g} (\mathbf {X}))\right)} ^ {\text {c a r r y g a t e}} \left(\sigma (\mathbf {g} (\mathbf {X})) \odot \mathbf {f} (\mathbf {X})\right) \tag {25} \\ \end{array}
$$
where the  $\sigma(.)$  can be seen as the transform gate, while  $(1 - \sigma(.))$  can be seen as the carry gate. This could be regarded as a form of highway networks.
3.3 Variant Gated Connections
Highway Gate Similar to the highway networks (Srivastava et al., 2015), let  $T(\mathbf{X})$  signal the transform gate and  $(1 - T(\mathbf{X}))$  be the carry gate, we have the highway-network-like structures by regulating the encoding  $f(\mathbf{X})$  with transform gate and controlling  $\mathbf{X}$  with carry gate. This is quite
6890
similar to highway networks:
$$
T (\mathbf {X}) = \sigma \left(\mathbf {X} \mathbf {W} _ {\mathbf {1}} + \mathbf {b} _ {\mathbf {1}}\right) \tag {26}
$$
$$
f (\mathbf {X}) = \mathbf {X} \mathbf {W} _ {2} + \mathbf {b} _ {2} \tag {27}
$$
$$
\begin{array}{l} o (\mathbf {X}) = (1 - T (\mathbf {X})) \odot \mathbf {X} \tag {28} \\ + T (\mathbf {X}) \odot f (\mathbf {X}) \\ \end{array}
$$
$$
\mathbf {U} = L N \left(o (\mathbf {X}) + \operatorname {A t t} (\mathbf {Q}, \mathbf {K}, \mathbf {V})\right) \tag {29}
$$
where Eq. 28 is the element-wise summation of highway networks,  $o(\cdot)$  represents the intermediate output.
Gated MHDPA Similar to previous highway gates, we can apply the carry gate and transform gate on the attention and FFN units respectively. Thus we have:
$$
\begin{array}{l} o (\mathbf {X}) = (1 - T (\mathbf {X})) \odot \operatorname {A t t} (\mathbf {Q}, \mathbf {K}, \mathbf {V}) \tag {30} \\ + T (\mathbf {X}) \odot f (\mathbf {X}) \\ \end{array}
$$
$$
\mathbf {U} = L N \left(o (\mathbf {X}) + \mathbf {X}\right) \tag {31}
$$
Such gates can be regarded as dynamically adjusting the information flow between the feature-wise representations and SANs (Eq. 30).
We apply the gating mentioned above on Transformer variants described in section 2 on LM tasks and respectively make comparisons in terms of both the convergence process and the final performance. For fairness, we apply SDU components based on the same hyperparameters as the original paper<sup>1</sup>. Our code is available<sup>2</sup>.
4.1 vs. Transformer / R-Transformer
We first evaluate the gating units on the Penn TreeBank (PTB) LM task. The SDU gates are added on Eq. 7, 9 for each Transformer block. All models in this section are trained on single NVIDIA Titan Xp GPU.
4.1.1 Char-level PTB
Hyperparameter and Training The gated components are evaluated on character-level PTB LM tasks (see Appendix A.1 for hyperparameter settings). The loss and bit per character (bpc) provide the metrics to evaluate the trained models. All models are trained with 100 epochs.
Figure 2: The 3-layer Transformer's curve of training and evaluation performance on character-level PTB LM.
Figure 3: The 3-layer RT's curve of training and evaluation performance on character-level PTB LM task.
Results of Transformer As shown in Table 1, all the gating-enhanced models conspicuously surpass the performance of the loss and perplexity over the baseline on both training and validating set, revealing the positive influence of self-gating units in supporting Transformer blocks. Furthermore, Fig. 2 presents the beneficial effect of gating units in accelerating the convergence process in both training and evaluation set by a clear margin, validating the accumulative effect that our gating units bring out. In which SDUs with tanh gates (8.76% improvement) outperform the counterpart with sigmoid gates (8.2% improvement) in terms of the final perplexity on the test set.
model	eval loss	eval ppl	test loss	test ppl
T-L3	1.068	1.541	1.036	1.495
+σ SDU	0.9776	1.410↓	0.950	1.371↓
+tanh SDU	0.9714	1.401↓	0.945	1.364↓
Table 1: Performance of 3 Layer Transformers and SDU components on char-level PTB LM task. The best performance is marked bold.
Results of RT It can be seen in Fig. 3 that supplementing SDUs can increase the speed of the convergence process of training and evaluation, strengthening our previous claim. As for the final perplexity on the test set,  $\sigma$ -gate SDUs could achieve better than baselines while tanh-gate SDUs perform a bit worse, as shown in Table 2. The influence of  $\sigma$ -gate SDUs might be owing to that  $\sigma$  function compresses the input into the dense non-zero ratios within  $(0,1)$  and results in stable variation range. In contrast, the zero-centered property and possibly
<sup>1</sup>Some results of baselines are slightly lower than those reported in original papers using the code obtained from authors but are within the limits of experimental error and variance. <sup>2</sup>https://github.com/cyk1337/ Highway-Transformer
6891
zeroed values of tanh may cause the corresponding units easier to be trapped into the premature convergence during the training process. Besides,  $\sigma$  gates have been empirically proved to be more stable than tanh gates in the follow-up experiments.
model	eval loss	eval ppl	test loss	test ppl
RT-L3	0.8896	1.283	0.867	1.250
+tanh SDU	0.9096	1.312	0.883	1.274
+σ SDU	0.8863	1.279↓	0.863	1.245↓
4.1.2 Word-level PTB
Hyperparameter and Training We compare the performance between 3-layer Transformer and R-Transformer (RT) with and without SDU gating units. Appendix A.1 illustrates the hyperparameter setup. All experiments are conducted with 100 epochs, and the loss and perplexity (ppl) values on the development set serve as evaluation metrics.
Figure 4: Loss and perplexity of 3-layer Transformers on the word-level PTB training and validation set.
Table 2: Performance of 3 Layer R-Transformers and SDU components on char-level PTB LM task.
model	eval loss	eval ppl	test loss	test ppl
T-L3	4.937	139.4	4.87	130.43
+σ SDU	4.934	138.9↓	4.87	130.30↓
+tanh SDU	5.001	148.5	4.94	139.53
Results of Transformer Figure 4 shows a noticeable downward trend on the evaluation performance (i.e., the validation loss and perplexity) of the attention model with tanh and sigmoid functions over the beginning 30 epochs, again indicating the convergence acceleration effect of our gated units. Also,  $\sigma$ -gate enhanced models outmatches the baseline on the test perplexity, but models with tanh gates reach into a plateau untimely. As for the training curves, Transformers with SDUs have
seen a remarkably sharper fall in comparison with the baseline model over all the training period.
Results of RT As in Fig. 5 and Table 4, models with SDUs entirely surpass the performance of the baseline involving both the convergence speed and perplexity on the test set. Similar to the word-level R-Transformer, tanh-gate SDUs behave a bit better than the counterpart with sigmoid gates, both showing stable curvatures of convergence.
Table 3: Performance of 3-layer basic Transformer (T-L3) and SDU components on word-level PTB LM.
model	eval loss	eval ppl	test loss	test ppl
RT-L3	4.58	97.63	4.53	92.31
+σ SDU	4.53	92.91↓	4.48	87.88↓
+tanh SDU	4.50	89.97↓	4.44	84.92↓
Table 4: The performance of 3-layer R-Transformers (RT-L3) and SDU components on word-level PTB LM.
Figure 5: Loss and perplexity of 3-layer RT on the word-level PTB training and validation sets.
4.2 Sub-total