SlowGuess/ACL_2020_OCR huggingface dataset:
To sum up, gating units have empirically expedited the convergence of Transformer blocks due to the enrichment of self-regulated features with skip-connections. It can be seen that  $\sigma$ -gate presents the stability to bear a hand to reach the plateau without hurting the test performance, but tanh-gate seems to be task- and data-dependent and could be better than  $\sigma$ -gate SDUs in some circumstances. We can see that our proposed gated units are complementary to the recurrent connections in RNNs and can boost the performance based on localRNN-encoded representations.
In the following experiment, we check whether it is necessary to apply gates on all the layers and probe the effect of SDU variants (i.e., "highway gate" and "gate MHDPA"). Due to the small size of PTB, we experiment on a larger LM dataset enwik8 and adopt the impressive Transformer-XL, one of the vital variant structures used in XLNet (Yang et al., 2019).
6892
4.3 vs. Transformer-XL
Hyperparameter See Appendix A.3 for detailed hyperparameter settings.
4.3.1 Results of 6-layer Transformer-XL
It is noticeable that Transformer-XL models with different gating variants all outperform the baseline with different margins in terms of both performance and convergence speed, as shown in Table 5. Fig. 6 shows that SDUs benefit the convergence and validation performance compared with baselines. Among which  $\sigma$ -gate SDUs ranked top by achieving  $3.1\%$  improvement of bpc on the dev set, followed by gates with tanh, gated MHDPA, highway gate with  $2.7\%$ ,  $1.8\%$ ,  $1.7\%$  advance respectively. We attribute such improvements to the augmented refined representations learned by our gated units, preventing the basic self-attention blocks from purely considering the contextual dependency. It is also illustrated that SDUs do not conflict with recurrence mechanisms in Transformer-XL.
Figure 6: The comparison between 6-layer Transformer-XL with adding different SDUs.
4.3.2 Ablation Study
6-layer Transformer-XL To probe whether it is required to augment SDUs on each Transformer layer, we supplement gates on layer 1-3, layer 3-6, and layer 1-6 but removing gates on FFN components (denoted " $\mathsf{FFN}$ " as in Table 5 (see Fig. 8 in Appendix B for detailed convergence curvatures). We find that supplementing tanh-gates on the bottom three layers contribute most to the overall performance while tanh-gates on the top three layers could hinder the test set performance. Low-level Transformer blocks can capture the information from localness while top layers usually focus on the global long-range dependency (Yang et al., 2018).
model	eval loss	eval bpc	test loss	test bpc
L6-XL	0.8843	1.276	0.86	1.24339
+tanh SDU	0.8602	1.241↓	0.84	1.21424↓
+σ SDU	0.8577	1.237↓	0.84	1.21123↓
+highway gate	0.8692	1.254↓	0.85	1.22177↓
+gated MHDPA	0.8682	1.253↓	0.85	1.22398↓
Ablation study
+tanh L1-6\FFN	0.8720	1.258↓	0.85	1.22866↓
+tanh L1-3	0.8660	1.249↓	0.85	1.22039↓
+tanh L3-6	0.8852	1.277↓	0.86	1.24420↓
+σ L1-6\FFN	0.8752	1.263↓	0.85	1.23332↓
+σ L1-3	0.8792	1.268↓	0.86	1.23589↓
+σ L3-6	0.8843	1.276↓	0.86	1.24261↓
Table 5: Results of 6-layer Transformer-XL (L6-XL) and augmented SDUs with different settings. “+σ” LI-6\FFN represents adding σ-SDUs on MHDPAs of 1-st to 6-th layers but not on FFN sublayers.
Thus gates on bottom layers could aid in learning syntax and superficial representations to some extent. It also indicates that our gates may be beneficial for the encoding of low-level fine-granularity representations rather than semantic meaning regulation on high-level layers.
12-layer Transformer-XL Previous experiments are all conducted on shallow models and illustrate the positive effects. To investigate the performance on deep stacked models, we further extend our trials to 12-layer Transformer-XL. All hyperparameters are the same as 6-layer Transformer-XL, as shown in Appendix A.3. Each Model is trained 400k steps for more than 100 hours on  $4 \times$  GeForce 2080Ti GPUs in parallel.
Figure 7: The comparison between 12-layer Transformer-XL with and without tanh gated units on bottom two layers.
The experimental results illustrate that SDU components have contributed to expediting the convergence during training (see Fig. 9 and 10 in Appendix C for details). But supplementing gated units on each Transformer block could encounter the premature convergence phenomenon. It is also observed that adding the bottom few layers with gated units could strengthen the convergence pro
6893
model	eval loss	eval bpc	test loss	test bpc
L12-XL	0.7554	1.090	0.74	1.07160
Ablation study
+tanh L1-12	0.7919	1.143	0.78	1.12797
+tanh L1-6	0.7623	1.100	0.75	1.08234
+tanh L1-3	0.7558	1.090	0.74	1.07140↓
+tanh L1-2	0.7548	1.089↓	0.74	1.06904↓
+tanh L1	0.7549	1.089↓	0.74	1.06960↓
+tanh L6-12	0.7572	1.092	0.74	1.07313
+tanh \FFN	0.7734	1.116	0.76	1.09920
+σ L1-12	0.7752	1.118	0.77	1.10462
+σ L1-6	0.7635	1.101	0.75	1.08283
+σ L1-3	0.7580	1.094	0.74	1.07383
+σ L1-2	0.7552	1.090	0.74	1.07148↓
+σ L1	0.7557	1.090	0.74	1.07157↓
+σ L6-12	0.7585	1.094	0.75	1.07607
+σ \FFN	0.7647	1.103	0.75	1.08652
+highway gate	0.7784	1.120	0.77	1.10922
+gated MHDPA	0.7741	1.117	0.76	1.10292
cess without impeding the final performance, as shown in Table 6. It is observed from Fig. 7 that tanh-gates on the bottom two layers promote the convergence process and further improve the bpc performance on the dev and test set.
Interestingly, the performance does not follow a positive correlation with the increase of gated layer numbers. We can see that enriching the bottom 2 layers with tanh and  $\sigma$  gated functions (denoted “+tanh L1-2” and “+σ L1-2” in Table 6) could impressively benefit for the convergence on both training and evaluation process and even marginally increase the final test bpc (see Fig. 9 and Fig. 10 in Appendix C for details). Therefore, the lower layers benefit more from our proposed gated units than higher layers, again illustrating that SDUs could enhance feature-wise information on shallow layers of deep-stacked Transformer components.
4.4 Gating Mechanism Analysis
It can be concluded that gating units could boost the convergence, especially on low-level layers. Enhancing the bottom layers of deep-stacked models may result in faster convergence of optimization. This may be owing to that SDU gates can enrich the original representations with adaptive self-dependency encodings. The final hidden state can be regarded as a revised representation that incorporating additional self-attentive features.
Meanwhile, we find that supplementing SDU gates does not increase much of the time cost in comparison with baselines. Instead, the total run
ning time of each experimental setting is quite similar. We summarize the training time costs of 6-layer Transformer-XL as table .7.
Table 6: Final results of 12-layer Transformer-XL (XL-L12) and augmented SDUs with different settings.
model	time cost (hour)
xl-L6	21.16
+tanh SDU	21.45
+σ SDU	21.87
+ highway gate	21.93
+gated MHDPA	21.10
Table 7: Summary of training time costs of 6-layer Transformer-XL.
It is argued that low-level transformers learn the local-region information while high-level layers pay more attention to global dependencies (Yang et al., 2018). Our experimental results could verify that gated representation on bottom layers can strengthen the performance by introducing additional gated encodings on localness.
Further, the visualization of learned gate bias parameters of 6-layer and 12-layer models, as shown in Fig. 11 in Appendix D.1, presenting the layer separation with the increase of layer depth. It has seamlessly verified our previous hypothesis that SDU on shallow layers could promote the learning process and attend to different information with top layers. The scatter plot of Fig. 12 in Appendix D.2 indicates that gates on different sublayers learn from different aspects in the identical representation space.
SDUs calculate the output by regulating the information flow of inputs conditioned on themselves. Given the hidden dimension of  $d$ , the additional cost of trainable parameters on each SDU unit in our experiments is  $O(2d(d + 1))$ . Meanwhile, convolutions along the sequence direction can substitute fully-connected feedforward SDU to curtail the extra parameter cost. Such gating units equip good scalability to attach to different Transformer structures with only minor modification of implementation.
The gradient of our SDU components is:
$$
\begin{array}{l} \nabla [ \mathbf {f} (\mathbf {x}) \odot \Psi (\mathbf {g} (\mathbf {x})) ] = \nabla \mathbf {f} (\mathbf {x}) \odot \Psi (\mathbf {g} (\mathbf {x})) (32) \\ + \mathbf {f} (\mathbf {x}) \odot \nabla \Psi (\mathbf {g} (\mathbf {x})) (33) \\ \end{array}
$$
where  $\mathbf{f},\mathbf{g}$  are linear projections and  $\Psi$  takes tanh or  $\sigma$  function. The addition operation of two terms provides an unimpeded information flow, which can be regarded as a multiplicative skip connection (Dauphin et al., 2017) while the second term is usually vanishing due to the derivative of the gating
6894
function  $\Psi$ . Based on the experimental results, we hypothesize that it could accelerate the optimization process to move towards a local minimum.
In recent years, there have been plenty of works adopting gating units into CNNs to help learn sequential information. Dauphin et al. (2017) proposed stacked gated CNNs by incorporating GLUs into the 1-dimensional convolution operation, achieving the competitive results in comparison to recurrent models on LM tasks. Based on this, Gehring et al. (2017) augmented the attention mechanism together with GLUs on the convolutional structures, also surpassing the deep LSTMs on NMT tasks. Recently, dynamic convolutions were used to replace MHDPA components in Transformers entirely and also get the impressive results on the WMT-14 dataset (Wu et al., 2019).
Amounts of works employed gating mechanisms to modulate self-attention sublayers. Gated-Attention Reader (Dhingra et al., 2016) introduced gated attention by computing gates on the query encoding to interact with document representations for reading comprehension. Zhang et al. (2018) replaced the first layer of Transformer decoding stacks with an average attention layer by computing forget gates using averaged preceding contextual encodings to regulate the current state information. Distance-based SAN (Im and Cho, 2017) and DiSAN (Shen et al., 2018) put a fusion gate to aggregate the representations after the multidimensional self-attention block for natural language inference. Lai et al. (2019) proposed a gated self-attention memory network with aggregated interactions between input sequences and context vectors for answer selection of question answering.
Notably, our SDU bears a resemblance to the activation Swish (Ramachandran et al., 2017) in terms of the equation format. Both of them use the sigmoidal function and self-gating mechanism. However, Swish controls the input gated on itself in a tandem way while the proposed SDU applies the gate after a linear projection and performs using a shunt connection in Transformer stacks.
Gating-enhanced architecture enjoys both the advantage of MHDPA and self-regulated gating mechanism, allowing for the pseudo-highway information flow for better convergence by elastically intro
ducing a few trainable parameters. It outperforms or matches the performance of common Transformer variants without hyperparameter tuning. It is empirically proved that self-gating units on shallow layers could provide more internal representations of importance and significantly benefit for convergence. This also supports the argument that different levels of Transformer components attend to different semantic aspects while lower levels pay more attention to local regions. In the future, it is necessary to interpret the semantics that Transformer layers in different depths can convey, which is beneficial for the computing-efficiency.
Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, and Llion Jones. 2019. Character-level language modeling with deeper self-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3159-3166.
Zihang Dai, Zhilin Yang, Yiming Yang, William W Cohen, Jaime Carbonell, Quoc V Le, and Ruslan Salakhutdinov. 2019. Transformer-xl: Attentive language models beyond a fixed-length context. arXiv preprint arXiv:1901.02860.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. 2017. Language modeling with gated convolutional networks. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 933-941. JMLR.org.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W Cohen, and Ruslan Salakhutdinov. 2016. Gated-attention readers for text comprehension. arXiv preprint arXiv:1606.01549.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning-Volume 70, pages 1243-1252. JMLR.org.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
6895
Jinbae Im and Sungzoon Cho. 2017. Distance-based self-attention network for natural language inference. arXiv preprint arXiv:1712.02047.
Tuan Lai, Quan Hung Tran, Trung Bui, and Daisuke Kihara. 2019. A gated self-attention memory network for answer selection. arXiv preprint arXiv:1909.09696.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. arXiv preprint arXiv:1806.00187.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. 2018. Image transformer. arXiv preprint arXiv:1802.05751.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assetss/researchcovers/languageeunsupervised/language understanding paper. pdf.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. 2017. Searching for activation functions. arXiv preprint arXiv:1710.05941.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018. Disan: Directional self-attention network for rnn/cnn-free language understanding. In Thirty-Second AAAI Conference on Artificial Intelligence.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015. Highway networks. arXiv preprint arXiv:1505.00387.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnicki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. 2019. Grandmaster level in starcraft ii using multi-agent reinforcement learning. Nature, pages 1-5.
Zhiwei Wang, Yao Ma, Zitao Liu, and Jiliang Tang. 2019. R-transformer: Recurrent neural network enhanced transformer. arXiv preprint arXiv:1907.05572.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. 2019. Pay less attention with lightweight and dynamic convolutions. arXiv preprint arXiv:1901.10430.
Baosong Yang, Zhaopeng Tu, Derek F Wong, Fandong Meng, Lidia S Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. arXiv preprint arXiv:1810.10182.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, et al. 2018. Deep reinforcement learning with relational inductive biases.
Biao Zhang, Deyi Xiong, and Jinsong Su. 2018. Accelerating neural transformer via an average attention network. arXiv preprint arXiv:1805.00631.
6896
A Experimental Setup Details
A.1 Hyperparameter Settings for RT on Char-level PTB
For RT on char-level PTB, we adopt the batch size of 16, gradient clipping with maximum L2 norm of 0.15, layer number of 3, hidden dimension of 512, the sequence length of 400 in char-level, the dropout rate for sublayer connection of 0.15, 8 heads for MHDPA, the initial learning rate of 2, SGD optimizer with linear decay, layer number of 3 in both Transformer and RT models. Weights are initialized with uniform distribution  $\mathbf{w} \sim U(-0.1, 0.1)$  and biases are all initialized as 0s. The size of GRU cells in localRNNs is set to 7 in RT.
A.2 Hyperparameter Settings for RT on Word-level PTB
We use the dropout rates of 0.35 and 0.15 for sublayer connections and word embeddings, the initial learning rate of 2, gradient clipping with the maximum L2 norm of 0.35, the hidden dimension of 128, 8-head attention, sequence length of 80 in both Transformer and RT. The weights are initialized with uniform distribution  $U(-0.01, 0.01)$ , and the biases are constant 0s. The optimizer is stochastic gradient descent (SGD) with annealed decay. The localRNN context size for LSTM cells is set to 9 in RT.
A.3 Hyperparameter Settings for Transformer-XL on enwik8
We use layer number of 6, 8 heads for MHDPA with hidden size 64 for each head, hidden size of 2,048 in FFN components, the dropout rate of 0.1 in FFN, embedding size of 512, learning rate 0.00025, memory length of 512, batch size of 22, Adam optimizer without the warm-up strategy. We initialize weights under the Gaussian  $\mathcal{N}(0,1)$  and biases as 0s.
B Experimental Results of 6-layer Transformer-XL
Fig 8 displays all the experimental curvatures with different SDU settings on 6-layer Transformer-XL.
Figure 8: The performance of 6-layer Transformer-XL experiments with various settings of gated units.
C Experimental Results of 12-layer Transformer-XL
C.1 Transformer-XL v.s. +tanh Gates
Fig. 9 shows the curve of tanh-gate enhanced Transformer-XL during the training and evaluation process. Adding tanh-gates on the first few layers greatly boost the convergence performance in both the training and evaluation process. Among which “+tanh L1-2” presents a rapid convergence trend and marginally outperforms the baseline performance.
Figure 9: The performance of 12-layer Transformer-XL experiments augmenting tanh gated units.
6897
C.2 Transformer-XL v.s. +sigmoid Gates
Fig. 10 illustrates the performance of Transformer-XL augmented with  $\sigma$  gates. The sigmoid-gated Transformer-XL has showed a similar trend as tanh gates in Fig. 9.
Figure 10: The performance of 12-layer Transformer-XL experiments augmenting  $\sigma$  gated units.
D Plot of Gate Biases of Transformer-XL
D.1 Heatmap Visualization
Fig 11 witnesses the visualization of learned biases, which are all initialized as zeros at the beginning. Obviously, the trainable biases of SDU gates perform quite different between on MHDPA and FFN sublayers as in Fig. 11a, 11c for 6-layer models and Fig. 11b, 11d for 12-layer models. Also, the gate biases are similarly distributed on all of the 6 layers, as in Fig. 11e, while showing the layer separation on the bottom few transformer layers as shown in Fig. 11f. This also verifies the experimental evidence that SDU gates on 6-layer models all positively influence the final test performance, but those only on the previous few layers of 12-layer transformers could have better results on both convergence speed and the final test bpc.
D.2 Scatter Visualization
Fig. 12 illustrates the uniform distribution on both 6-layer and 12-layer Transformer-XL models. Due to the existence of residual connections, the representation space can be seen as the same. Hence the evenly distributed gate biases may learn from
different aspects accordingly, which also matches our common intuition.
6898
(a) Plot of gate biases on MHDPA of 6-layer models.
(b) Plot of gate biases on MHDPA of 12-layer models.
(c) Plot of gate biases on FFN of 6-layer models.
(d) Plot of gate biases on FFN of 12-layer models.
(e) Plot of gate biases on all sublayers of 6-layer models.
(f) Plot of gate biases on all sublayers of 12-layer models.
Figure 11: The heatmap visualization of learnable biases (i.e.,  $\mathbf{b}_{\mathbf{l}}$  in Eq. 21) on  $\sigma$  gate units of 6-layer (left column) and 12-layer (right column) Transformer-XL models, where vertical axes represent the layer number of our models, and "a1" and "b3" denote the 1-st MHDPA sublayer and 3-rd FFN sublayer, respectively. All gate biases are initialized as 0s with 512 dimension of each.
6899
(a) Plot of bias distributions on 6-layer models.
(b) Plot of bias distributions on 12-layer models.
Figure 12: Scatter visualization of SDU gate biases on 6-layer and 12-layer Transformer-XL, where "layer2-SA" denotes the gate bias on 2-nd self-attention sublayer. We employ t-Distributed Stochastic Neighbor Embedding (t-SNE) to reduce the dimension from 512 to 2.