SlowGuess/ACL_2020_OCR huggingface dataset:
5 Evaluation of Structure Modeling
In this section, we investigate whether SSANs better capture structural information of sentences. To this end, we first empirically evaluate the syntactic structure knowledge embedded in the learned representations (§5.1). Then we investigate the attention behaviors by extracting constituency tree from the attention distribution (§5.2).
$^{6}$ https://github.com/baosongyang/WRD.
Class	Ratio	SANs	SSANs	△
5	6.9%	68.66	75.22	+9.6%
6	14.3%	56.10	64.09	+14.2%
7	16.3%	46.63	55.05	+18.1%
8	17.9%	39.68	50.88	+28.2%
9	17.4%	38.33	50.97	+33.0%
10	15.3%	35.54	49.88	+40.3%
11	11.9%	48.86	56.39	+15.4%
All	100%	45.68	55.90	+22.4%
Table 4: F1 score on the tree depth task. "Ratio" denotes the portion each class takes.
Type	Ratio	SANs	SSANs	\( \bigtriangleup \)
Ques.	10%	95.90	97.06	+1.2%
Decl.	60%	88.48	91.34	+3.2%
Clau.	25%	72.78	78.32	+7.6%
Other	5%	50.67	61.13	+20.6%
All	100%	83.78	87.25	+4.1%
5.1 Structures Embedded in Representations
Task Description We leverage two linguistic probing tasks to assessing the syntactic information embedded in a given representation. Both tasks are cast as multi-label classification problem based on the representation of a given sentence, which is produced by an examined model:
Tree Depth (TreeDepth) task (Conneau et al., 2018) checks whether the examined model can group sentences by the depth of the longest path from root to any leaf in their parsing tree. Tree depth values range from 5 to 11, and the task is to categorize sentences into the class corresponding to their depth (7 classes).
Top Constituent (TopConst) task (Shi et al., 2016) classifies the sentence in terms of the sequence of top constituents immediately below the root node, such as "ADVP NP VP ". The top constituent sequences fall into 20 categories: 19 classes for the most frequent top constructions, and one for all other constructions.
We trained the model on the open-source dataset provided by Conneau et al. (2018), and used the same model architecture in Section 4.1.
Probing Accuracy Table 4 lists the results on the TreeDepth task. SSANs significantly outper
Table 5: F1 score on the top constituent task. We report detailed results on 4 types of sentences: question ("Ques.", declarative ("Decl.", a clause ("Clau.",nd other ("Other") sentences.
Metric	SANs	SSANs	△
BP	21.09	22.07	+4.7%
BR	22.05	23.07	+4.6%
F1	-21.56	-22.56	+4.2%
Table 6: Evaluation on constituency trees generated from the attention distribution.
form SANs by  $22.4\%$  on the overall performance. Concretely, the performance of SANs dramatically drops as the depth of the sentences increases. $^{7}$  On the other hand, SSANs is more robust to the depth of the sentences, demonstrating the superiority of SSANs on capturing complex structures.
Table 5 shows the results on the TopConst task. We categorize the 20 classes into 4 categories based on the types of sentences: question sentence ("* SQ ."), declarative sentence ("* NP VP *" etc.), clause sentence ("SBAR *" and "S *"), and others ("OTHER"). Similarly, the performance of SANs drops as the complexity of sentence patterns increases (e.g., "Ques."  $\Rightarrow$  "Others",  $95.90\Rightarrow 50.67$ ). SSANs significantly improves the prediction F1 score as the complexity of sentences increases, which reconfirm the superiority of SSANs on capturing complex structures.
5.2 Structures Modeled by Attention
Task Description We evaluate the ability of self-attention on structure modeling by constructing constituency trees from the attention distributions. Under the assumption that attention distribution within phrases is stronger than the other, Mareček and Rosa (2018) define the score of a constituent with span from position  $i$  to position  $j$  as the attention merely inside the span denoted as  $\text{score}(i,j)$ . Based on these scores, a binary constituency tree is generated by recurrently splitting the sentence. When splitting a phrase with span  $(i,j)$ , the target is to look for a position  $k$  maximizing the scores of the two resulting phrases:
$$
k = \underset {k ^ {\prime}} {\arg \max } \left(\operatorname {s c o r e} \left(i, k ^ {\prime}\right) \cdot \operatorname {s c o r e} \left(k ^ {\prime}, j\right)\right) \tag {6}
$$
We utilized Stanford CoreNLP toolkit to annotate English sentences as golden constituency trees. We used EVALB<sup>8</sup> to evaluate the generated constituency trees, including bracketing precision, bracketing recall, and bracketing F1 score.
The only exception is the class of "11", which we attribute to the extraction of feature of associating "very complex sentence" with maximum depth "11".
$^{8}$ http://nlp.cs.nyu.edu/evalb.
(a) SANs
(b) SSANs
Figure 6: Example of constituency trees generated from the attention distributions.
Type	TreeDepth	TopConst	En⇒De Translation
SANs	SSANs	\( \bigtriangleup \)	SANs	SSANs	\( \bigtriangleup \)	SANs	SSANs	\( \bigtriangleup \)
Content	Noun	0.149	0.245	+64.4%	0.126	0.196	+55.6%	0.418	0.689	+64.8%
Verb	0.165	0.190	+15.2%	0.165	0.201	+21.8%	0.146	0.126	-13.7%
Adj.	0.040	0.069	+7.3%	0.033	0.054	+63.6%	0.077	0.074	-3.9%
Total	0.354	0.504	+42.4%	0.324	0.451	+39.2%	0.641	0.889	+38.7%
Content-Free	Prep.	0.135	0.082	-39.3%	0.123	0.119	-3.3%	0.089	0.032	-64.0%
Dete.	0.180	0.122	-32.2%	0.103	0.073	-29.1%	0.070	0.010	-85.7%
Punc.	0.073	0.068	-6.8%	0.078	0.072	-7.7%	0.098	0.013	-86.7%
Others	0.258	0.224	-13.2%	0.373	0.286	-23.3%	0.102	0.057	-41.1%
Total	0.646	0.496	-23.3%	0.676	0.549	-18.8%	0.359	0.111	-69.1%
Table 7: Attention distributions on linguistic roles for the structure modeling probing tasks (§5.1, "TreeDepth" and "TopConst") and the constituency tree generation task (§5.2, "En⇒De Translation").
Parsing Accuracy As shown in Table 6, SSANs consistently outperform SANs by  $4.6\%$  in all the metrics, demonstrating that SSANs better model structures than SANs. Figure 6 shows an example of generated trees. As seen, the phrases "he ran" and "heart pumping" can be well composed for both SANs and SSANS. However, SANs fail to parse the phrase structure "legs churning", which is correctly parsed by SSANs.
5.3 Analysis on Linguistic Properties
In this section, we follow He et al. (2019) to analyze the linguistic characteristics of the attended words in the above structure modeling tasks, as listed in Table 7. Larger relative increase ("△") denotes more attention assigned by SSANs. Clearly, SSANs pay more attention to content words in all cases, although there are considerable differences among NLP tasks.
Content words possess semantic content and contribute to the meaning of the sentence, which are essential in various NLP tasks. For example, the depth of constituency trees mainly relies on the nouns, while the modifiers (e.g., adjective and
content-free words) generally make less contributions. The top constituents mainly consist of VP (95% examples) and NP (75% examples) categories, whose head words are verbs and nouns respectively. In machine translation, content words carry essential information, which should be fully transformed to the target side for producing adequate translations. Without explicit annotations, SANs are able to learn the required linguistic features, especially on the machine translation task (e.g., dominating attention on nouns). SSANs further enhance the strength by paying more attention to the content words.
However, due to their high frequency with a limited vocabulary (e.g., 150 words $^9$ ), content-free words, or function words generally receive a lot of attention, although they have very little substantive meaning. This is more series in structure probing tasks (i.e., TreeDepth and TopConst), since the scalar guiding signal (i.e., class labels) for a whole sentence is non-informative as it does not necessarily preserve the picture about the intermediate syntactic structure of the sentence that is being
$^{9}$ https://en.wikipedia.org/wiki/Function_word.
generated for the prediction. On the other hand, the problem on content-free words is alleviated on machine translation tasks due to the informative sequence signals. SSANs can further alleviate this problem in all cases with a better modeling of structures.
In this work, we make an early attempt to assess the strengths of the selective mechanism for SANs, which is implemented with a flexible Gumbel-Softmax approach. Through several well-designed experiments, we empirically reveal that the selective mechanism migrates two major weaknesses of SANs, namely word order encoding and structure modeling, which are essential for natural language understanding and generation. Future directions include validating our findings on other SAN architectures (e.g., BERT (Devlin et al., 2019)) and more general attention models (Bahdanau et al., 2015; Luong et al., 2015).
We thank the anonymous reviewers for their insightful comments. We also thank Xiaocheng Feng, Heng Gong, Zhangyin Feng, and Xiachong Feng for helpful discussion. This work was supported by the National Key R&D Program of China via grant 2018YFB1005103 and National Natural Science Foundation of China (NSFC) via grant 61632011 and 61772156.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP.
Mia Xu Chen, Orhan Firat, Ankur Bapna, Melvin Johnson, Wolfgang Macherey, George Foster, Llion Jones, Mike Schuster, Noam Shazeer, Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Zhifeng Chen, Yonghui Wu, and Macduff Hughes. 2018. The best of both worlds: Combining recent advances in neural machine translation. In ACL.
Alexis Conneau, German Kruszewski, Guillaume Lample, Loïc Barrault, and Marco Baroni. 2018. What You Can Cram Into A Single Vector: Probing Sentence Embeddings for Linguistic Properties. In ACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Xiaocheng Feng, Ming Liu, Jiahao Liu, Bing Qin, Yibo Sun, and Ting Liu. 2018. Topic-to-essay generation with neural networks. In *IJCAI*.
Xiaocheng Feng, Duyu Tang, Bing Qin, and Ting Liu. 2016. English-chinese knowledge base translation with neural network. In COLING.
Emil Julius Gumbel. 1954. Statistical theory of extreme values and some practical applications: a series of lectures. U. S. Govt. Print. Office.
Maosheng Guo, Yu Zhang, and Ting Liu. 2019. Gaussian transformer: a lightweight approach for natural language inference. In AAAI.
Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. 2019a. Multi-granularity self-attention for neural machine translation. In EMNLP.
Jie Hao, Xing Wang, Shuming Shi, Jinfeng Zhang, and Zhaopeng Tu. 2019b. Towards better modeling hierarchical structure for self-attention with ordered neurons. In EMNLP.
Jie Hao, Xing Wang, Baosong Yang, Longyue Wang, Jinfeng Zhang, and Zhaopeng Tu. 2019c. Modeling recurrence for transformer. In NAACL.
Shilin He, Zhaopeng Tu, Xing Wang, Longyue Wang, Michael Lyu, and Shuming Shi. 2019. Towards understanding neural machine translation with word importance. In EMNLP.
Tianyu He, Xu Tan, Yingce Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. 2018. Layer-wise coordination between encoder and decoder for neural machine translation. In NIPS.
Xiaochen Hou, Jing Huang, Guangtao Wang, Kevin Huang, Xiaodong He, and Bowen Zhou. 2019. Selective attention based graph convolutional networks for aspect-level sentiment classification. ArXiv, abs/1910.10857.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with gumbel-softmax. In ICLR.
Jian Li, Zhaopeng Tu, Baosong Yang, Michael R. Lyu, and Tong Zhang. 2018. Multi-Head Attention with Disagreement Regularization. In EMNLP.
Jian Li, Baosong Yang, Zi-Yi Dou, Xing Wang, Michael R. Lyu, and Zhaopeng Tu. 2019. Information aggregation for multi-head attention with routing-by-agreement. In NAACL.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. In ICLR.
Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective Approaches to Attention-based Neural Machine Translation. In EMNLP.
Chris J Maddison, Daniel Tarlow, and Tom Minka. 2014. A* sampling. In NIPS.
David Mareček and Rudolf Rosa. 2018. Extracting syntactic trees from transformer encoder self- attentions. In BlackboxNLP@EMNLP.
Graham Neubig. 2011. The Kyoto free translation task. http://www.phontron.com/kftt.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In EMNLP.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep Contextualized Word Representations. In NAACL.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Hwee Tou Ng, Anders Björkelund, Olga Uryupina, Yuchen Zhang, and Zhi Zhong. 2013. Towards robust linguistic analysis using ontonotes. In CoNLL.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In ACL.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-Attention with Relative Position Representations. In NAACL.
Chen Shen, Guo-Jun Qi, Rongxin Jiang, Zhongming Jin, Hongwei Yong, Yaowu Chen, and Xian-Sheng Hua. 2018a. Sharp attention network via adaptive sampling for person re-identification. IEEE Transactions on Circuits and Systems for Video Technology, 29(10):3016-3027.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Shirui Pan, and Chengqi Zhang. 2018b. DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding. In AAAI.
Tao Shen, Tianyi Zhou, Guodong Long, Jing Jiang, Sen Wang, and Chengqi Zhang. 2018c. Reinforced self-attention network: a hybrid of hard and soft attention for sequence modeling. In *IJCAI*.
Xing Shi, Inkit Padhi, and Kevin Knight. 2016. Does String-based Neural MT Learn Source Syntax? In EMNLP.
Kaitao Song, Xu Tan, Di He, Jianfeng Lu, Tao Qin, and Tie-Yan Liu. 2018. Double path networks for sequence to sequence learning. In COLING.
Emma Strubell, Patrick Verga, Daniel Andor, David Weiss, and Andrew McCallum. 2018. Linguistically-informed self-attention for semantic role labeling. In EMNLP.
Zhixing Tan, Mingxuan Wang, Jun Xie, Yidong Chen, and Xiaodong Shi. 2018. Deep semantic role labeling with self-attention. In AAAI.
Gongbo Tang, Mathias Müller, Annette Rios, and Rico Sennrich. 2018. Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures. In EMNLP.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All You Need. In NIPS.
Xing Wang, Zhaopeng Tu, Longyue Wang, and Shuming Shi. 2019a. Self-attention with structural position representations. In EMNLP.
Yaushian Wang, Hung-Yi Lee, and Yun-Nung Chen. 2019b. Tree transformer: Integrating tree structures into self-attention. In EMNLP.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, pages 229-256.
Wei Wu, Houfeng Wang, Tianyu Liu, and Shuming Ma. 2018. Phrase-level self-attention networks for universal sentence encoding. In EMNLP.
Baosong Yang, Zhaopeng Tu, Derek F. Wong, Fandong Meng, Lidia S. Chao, and Tong Zhang. 2018. Modeling localness for self-attention networks. In EMNLP.
Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019a. Assessing the ability of self-attention networks to learn word order. In ACL.
Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, and Zhaopeng Tu. 2019b. Convolutional self-attention networks. In NAACL.
Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguuo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. 2019c. Modeling point clouds with self-attention and gumbel subset sampling. In CVPR.