SlowGuess/NAACL_2021_OCR huggingface dataset:
The results of the three tree-generation algorithms are shown in Table 2, 3 and 4 along with the performance of a random baseline obtained by running the algorithms on 10 random matrices. Here, we present the results of three selected models, limited to the performance of the first two layers for the 6-layer models, to allow for a direct comparison to the 2-layer models<sup>5</sup>. Across evaluations, the layer-wise performance within the same models are rather distinct, indicating that different properties are learned in the layers. This finding is in line with previous work (Liu et al., 2019a), especially
Complete evaluation results for all six models are presented in Appendix A.
Results for all six models can be found in Appendix B.
Model	No Cons.	Sent Cons.
Attn#0	Attn#1	Attn#0	Attn#1
RST-DT
CNNDM-2-1	61.2	59.7	76.2	74.6
CNNDM-6-8	60.3	60.8	75.4	75.0
NYT-6-8	62.4	62.2	76.7	75.6
Random	58.6±0.1	74.1±0.1
Instruction
CNNDM-2-1	61.1	59.8	71.4	↓70.3
CNNDM-6-8	60.3	61.2	71.2	70.9
NYT-6-8	61.3	61.3	71.3	↓70.0
Random	59.5±0.3	70.5±0.1
GUM
CNNDM-2-1	58.7	57.7	72.7	71.9
CNNDM-6-8	58.9	59.3	72.4	72.7
NYT-6-8	59.6	59.3	72.2	71.6
Random	57.5±0.1	71.5±0.2
given that the performance of each layer is consistent across constituency and dependency parsing outputs for all datasets. Furthermore, the more layers the summarization model contains, the smaller the performance gap between layers becomes. We believe that this could be caused by the discourse information being further spread across different layers. Generally, we observe that models trained on the CNNDM dataset perform better than models trained on the NYT corpus, despite the larger size of the NYT dataset. Plausibly, the superior performance of our models trained on CNNDM potentially reflects a higher diversity within documents in the CNNDM dataset.
Comparing the constituency tree performance in Table 2 against the dependency tree results in Tables 3 and 4, we can clearly see that the improvement of the constituency parsing approach over the random baseline is much smaller than the improvements for the generated dependency parse-trees. Presumably, this larger improvement for the dependency trees is due to the fact that dependency relationships (strongly encoding the nuclearity attribute) are more directly related to the summarization task than the plain structure information. This is in line with previous work on applying dependency trees to the summarization task (Hirao et al., 2013; Xu et al., 2020) and indicates that the learned attention matrices contain valid discourse information.
Table 2: RST Parseval Scores of generated constituency trees on the three datasets, expressed as 'Avg.  $\pm$  Std'. Green means the result is better than Random, and Red along with  $\downarrow$  means worse. Results for Random are obtained by applying the parser to random matrices for 10 times. Attn#0/1 are the first two layers.
Model	No Cons.	Sent Cons.
Attn#0	Attn#1	Attn#0	Attn#1
RST-DT
CNNDM-2-1	23.7	↓4.8	28.2	↓18.2
CNNDM-6-8	↓7.9	20.5	↓13.8	27.8
NYT-6-8	15.7	12.5	24.3	↓18.9
Random	11.2±0.2	20.3±0.2
Instruction
CNNDM-2-1	31.1	↓4.4	29.3	↓13.5
CNNDM-6-8	↓8.5	19.5	↓9.9	22.0
NYT-6-8	16.2	↓12.1	22.8	↓16.4
Random	13.1±0.3	19.3±0.4
GUM
CNNDM-2-1	21.3	↓2.24	27.3	↓16.1
CNNDM-6-8	↓4.7	15.8	↓11.5	24.80
NYT-6-8	12.6	↓9.6	23.4	↓17.1
Random	10.4±0.2	19.2±0.3
Table 3: Unlabeled Attachment Scores of dependency trees generated by the Eisner algorithm.
Model	No Cons.	Sent Cons.
Attn#0	Attn#1	Attn#0	Attn#1
RST-DT
CNNDM-2-1	21.6	↓1.5	29.3	19.6
CNNDM-6-8	7.3	17.3	↓16.1	28.5
NYT-6-8	13.7	10.6	25.0	21.1
Random	1.7±0.1	18.7±0.1
Instruction
CNNDM-2-1	28.1	↓2.1	37.4	18.1
CNNDM-6-8	6.9	15.9	↓14.9	25.8
NYT-6-8	14.8	9.8	25.4	21.1
Random	2.9±0.2	17.9±0.4
GUM
CNNDM-2-1	19.5	↓0.7	28.8	17.9
CNNDM-6-8	4.0	13.1	↓14.9	25.4
NYT-6-8	10.7	8.2	23.0	19.5
Random	0.9±0.05	17.0±0.2
Table 4: Unlabeled Attachment Scores of dependency trees generated by the CLE algorithm
As for the two approaches to dependency parsing, although Eisner generally outperforms CLE, the improvement over random trees is larger for CLE. We believe that this effect is due to the reduced constraints imposed on the CLE algorithm, which is not limited to generate projective trees.
Considering all three methods, the results of the CLE-generated dependency tree seem most promising. A possible explanation is that both CKY and Eisner build the discourse tree in a bottom-up fashion with dynamic programming. This way, only local information is used on lower levels of the tree. On the other hand, the CLE algorithm uses global information, potentially more aligned with the summarization task, where all EDUs are considered to predict importance scores.
Figure 4: The Unlabeled Attachment Score of trees generated by the attention matrix per head on three datasets under two conditions with the CNNDM-6-8 model.
4.5 Performance of Heads
While all previous results rely on the average attention matrices, we now analyze whether discourse information is evenly distributed across attention heads, or if a subset of the heads contains the majority of discourse related information.
We describe this analysis only for CLE for two reasons: (a) the summarization model seemingly captures more dependency-related discourse information than structure information; (b) compared with Eisner, the CLE approach is more flexible, by also covering non-projective dependency trees.
Since the results across all summarization models are consistent, we only show the accuracy heatmap for the CNNDM-6-8 model on the three RST-style discourse datasets in Figure 4. Remarkably, for all three datasets, there is one head in the model capturing the vast majority of discourse information, especially in the unconstrained case. Furthermore, the performance of the best single attention head is much better than the one of the average attention matrix shown in section 4.4 (e.g. 34.53 compared to 19.51 on the GUM dataset without sentence constraints). These intriguing findings will be further explored in future work.
4.6 Analysis of Generated Trees
Localness of Trees: To further verify that the generated trees are non-trivial, for instance simply connecting adjacent EDUs, we analyze the quality of the trees produced with the second attention head on the second layer, which is the top performer among all the heads shown in Figure 4. First, we separate all dependency relationships into two classes: local, holding between two adjacent EDUs, and distant, including all other relations between non-adjacent EDUs. Then we compute the ratio of the correctly predicted dependencies which are local (Local Ratio Corr.), as well as the
Measurement(%)	No Cons.	Sent Cons.
RST-DT
Local Ratio Corr.	77.78	79.17
Local Ratio GT	53.22
Local Ratio Ours	46.52	58.35
Instruction
Local Ratio Corr.	81.15	84.90
Local Ratio GT	59.82
Local Ratio Ours	47.90	60.54
GUM
Local Ratio Corr.	77.99	80.20
Local Ratio GT	53.28
Local Ratio Ours	39.97	53.76
Table 5: Measurements on the locality of the generated dependency trees, all numbers are in %. Corr. represents all the correct predictions, GT the ground-truth trees, and Ours the generated tree respectively.
ratio of local dependencies in the generated trees (Local Ratio Ours), and in the ground-truth trees (Local Ratio GT). The results of this analysis are shown in Table 5. For all datasets, the ratio of correctly predicted local dependencies (Local Ratio Corr.) (being  $>50$ ) is larger than the ratio for distant relations, which appears reasonable, since local dependency predictions are easier to predict than distant ones. Further, comparing (Local Ratio GT) and (Local Ratio Ours) without the sentence constraint (first column) shows that the number of local dependency relations in the ground-truth discourse trees is consistently larger than the predicted number. This indicates that the discourse information learned in the attention matrices goes beyond the oftentimes predominant local positional information. However, even without the sentence constraint (first column), when the CLE algorithm can predict trees of any form, more than  $40\%$  of the relations are predicted as local, suggesting that the standard CLE approach can already capture local information well.
Adding the sentence constraint (second column),
Branch	Height	Leaf	Arc	vac. (%)
RST-DT
Ours(Sent Cons)	1.50	27.06	0.37	0.10	3%
Ours(No Cons)	1.74	25.76	0.49	0.12	3%
GT Tree	2.10	8.19	0.51	0.13	2%
Instruction
Ours(Sent Cons)	1.56	15.74	0.39	0.13	3%
Ours(No Cons)	1.80	14.35	0.50	0.14	3%
GT Tree	1.59	8.49	0.41	0.15	1%
GUM
Ours(Sent Cons)	1.61	44.94	0.40	0.05	0%
Ours(No Cons)	2.14	43.08	0.54	0.08	0%
GT Tree	2.02	12.17	0.51	0.04	0%
Table 6: Statistics of our generated trees and the gold standard trees in terms of the average branch width, average height, average leaf ratio (micro), average normalized arc length of the trees and percentage of the Vacuous trees.
we find that the local dependency ratio of the generated trees (Local Ratio Ours) further increases by more than  $10\%$  across all three datasets. This makes intuitive sense, since the sentence constraint forces the generated trees to purely focus on local aspects within each sentence. To sum up, we find that the learned attention matrices contains both local and distant dependency information, although local dependency predictions perform better.
Properties of Trees: Following Ferracane et al. (2019), we structurally inspect the generated dependency trees, and compare them with the gold trees on all three datasets. This comparison is presented in Table 6, showing the average branch width, average height, average leaf ratio (micro) and average normalized arc length of the trees as well as the percentage of vacuous trees in each dataset<sup>6</sup>.
Looking at Table 6, it appears that our tree structure properties are similar to the ground-truth properties in regards to all measures except the height of the tree, which indicates that our trees tend to be generally deeper than gold standard trees, despite having a similar branch width and leaf ratio. Furthermore, our trees are even deeper when using the sentence constraint. Plausibly, by forcing each sentence to have its own sub-tree can make shallower inter-sentential structures less likely. Exploring potential causes for the difference in tree-height, possibly due to the summarization task itself, are left as future work.
4.7 Additional Results on Model Sensitivity to Initialization and Summarizer Quality
To investigate whether the performance is consistent cross different random initializations, and to explore the influence of the results with respect to the quality of the summarizer, we perform additional experiments with the 'CNNDM-6-8' model<sup>7</sup>. Overall, we find that the performance is rather similar across random initializations. Interestingly, a single head consistently shows better performance than all other heads across different initialization as well as datasets; however, while the position of the top-performing head is not always the same, it is often located in the second layer of the model. Regarding the second experiment exploring sensitivity to the summarizer quality, we create summarizers of increasing quality by providing more and more training. As expected, we find that as the summarization model is trained for additional steps, more accurate discourse information is learnt, concentrated in a single head.
We present a novel framework to infer discourse trees from the attention matrices learned in a transformer-based summarization model. Experiment across models and DATs indicates that both dependency and structural discourse information are learned, that such information is typically concentrated in a single head, and that the attention matrix also covers long distance discourse dependencies. Overall, consistent results across datasets and models suggest that the learned discourse information is general and transferable inter-domain.
In the future, we want to explore if simpler summarizers like BERTSUM (Liu and Lapata, 2019b) can also capture discourse info; specifically studying if the importance of the heads corresponds to the captured discourse info, which may help pruning summarization model by incorporating discourse info, in spirit of Xiao et al. (2020).
With respect to dependency tree generation possible improvements could come by looking for additional strategies balancing between guidance and flexibility, as Kuhlmann and Nivre (2006) explore for syntactic dependency parsing.
To address the problem of data sparsity in discourse parsing, we want to synergistically leverage other discourse-related tasks, in addition to sentiment and summarization, like topic modeling.
7More details can be found in Appendix C.
${}^{6}$  A vacuous tree is a special tree in which the root is one of the first two EDUs, with all nodes are children of the root.
We thank the anonymous reviewers and the UBC-NLP group for their insightful comments. This research was supported by the Language & Speech Innovation Lab of Cloud BU, Huawei Technologies Co., Ltd.
We further acknowledge the support of the Natural Sciences and Engineering Research Council of Canada (NSERC).
Lynn Carlson, Mary Ellen Okuowski, and Daniel Marcu. 2002. RST discourse treebank. Linguistic Data Consortium, University of Pennsylvania.
Y. Chu and T. Liu. 1965. On the shortest arborescence of a directed graph.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. *NAACL HLT* 2018 - 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies - Proceedings of the Conference, 2:615-621.
Jack Edmonds. 1967. Optimum Branchings.
Jason M. Eisner. 1996. Three new probabilistic models for dependency parsing: An exploration. In COLING 1996 Volume 1: The 16th International Conference on Computational Linguistics.
Elisa Ferracane, Greg Durrett, Junyi Jessy Li, and Katrin Erk. 2019. Evaluating discourse in structured text representations. arXiv preprint arXiv:1906.01472.
Tsutomu Hirao, Yasuhisa Yoshida, Masaaki Nishino, Norihto Yasuda, and Masaaki Nagata. 2013. Single-Document Summarization as a Tree Knapsack Problem. Technical report.
Patrick Huber and Giuseppe Carenini. 2019. Predicting discourse structure using distant supervision from sentiment. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2306-2316.
Patrick Huber and Giuseppe Carenini. 2020. MEGA RST discourse treebanks with structure and nuclearity from scalable distant sentiment supervision. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7442-7457, Online. Association for Computational Linguistics.
Shafiq Joty, Giuseppe Carenini, and Raymond T Ng. 2015. Codra: A novel discriminative framework for rhetorical analysis. Computational Linguistics, 41(3):385-435.
Dan Jurafsky and James H Martin. 2014. Speech and language processing, volume 3. Pearson London.
Yuta Kikuchi, Tsutomu Hirao, Hiroya Takamura, Manabu Okumura, and Masaaki Nagata. 2014. Single document summarization based on nested tree structure. 52nd Annual Meeting of the Association for Computational Linguistics, ACL 2014 - Proceedings of the Conference, 2:315-320.
Naoki Kobayashi, Tsutomu Hirao, Kengo Nakamura, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2019. Split or merge: Which is better for unsupervised rst parsing? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5801-5806.
Marco Kuhlmann and Joakim Nivre. 2006. Mildly non-projective dependency structures. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 507-514, Sydney, Australia. Association for Computational Linguistics.
Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014. Text-level discourse dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25-35, Baltimore, Maryland. Association for Computational Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2018. Learning structured text representations. Transactions of the Association for Computational Linguistics, 6:63-75.
Yang Liu and Mirella Lapata. 2019a. Hierarchical transformers for multi-document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5070-5081, Florence, Italy. Association for Computational Linguistics.
Yang Liu and Mirella Lapata. 2019b. Text Summarization with Pretrained Encoders. EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference, pages 3730-3740.
Yang Liu, Ivan Titov, and Mirella Lapata. 2019b. Single document summarization as tree induction. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1745-1755, Minneapolis, Minnesota. Association for Computational Linguistics.
William C. Mann and Sandra A. Thompson. 1988. Rhetorical Structure Theory: Toward a functional theory of text organization.
Daniel Marcu. 1999. Discourse Trees are Good Indicators of Importance in Text. Advances in Automatic Text Summarization, pages 123-136.
David Mareček and Rudolf Rosa. 2019. From balustrades to pierre vinken: Looking for syntax in transformer self-attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 263–275, Florence, Italy. Association for Computational Linguistics.
Ryan McDonald. 2007. A study of global inference algorithms in multi-document summarization. In Proceedings of the 29th European Conference on IR Research, ECIR'07, page 557-564, Berlin, Heidelberg. Springer-Verlag.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulíçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Noriki Nishida and Hideki Nakayama. 2020. Unsupervised discourse constituency parsing using viterbi em. Transactions of the Association for Computational Linguistics, 8:215-230.
Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287-297, Brussels, Belgium. Association for Computational Linguistics.
Evan Sandhaus. 2008. The new york times annotated corpus.
Rajen Subba and Barbara Di Eugenio. 2009. An effective discourse parser that uses rich linguistic information. In Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 566-574. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz
Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in Neural Information Processing Systems, 2017-Decem(Nips):5999-6009.
Chenguang Wang, Xiao Liu, and Dawn Song. 2020. Language models are open knowledge graphs.
Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184-188.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166-4176, Online. Association for Computational Linguistics.
Wen Xiao and Giuseppe Carenini. 2019. Extractive summarization of long documents by combining global and local context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3011-3021, Hong Kong, China. Association for Computational Linguistics.
Wen Xiao, Patrick Huber, and Giuseppe Carenini. 2020. Do we really need that many parameters in transformer for extractive summarization? discourse can help! In Proceedings of the First Workshop on Computational Approaches to Discourse, pages 124-134, Online. Association for Computational Linguistics.
Jiacheng Xu, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Discourse-aware neural extractive text summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5021-5031, Online. Association for Computational Linguistics.
Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition-based neural rst parsing with implicit syntax features. In Proceedings of the 27th International Conference on Computational Linguistics, pages 559-570.
Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. Language Resources and Evaluation, 51(3):581-612.
Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5059-5069, Florence, Italy. Association for Computational Linguistics.
A Performance of the Summarization Task
Table.7 shows the performance of different summarization models. In general, adding additional layers and heads does not consistently increase the performance on the summarization task itself.
Dataset	#Layer	#Head	R-1	R-2	R-L
CNNDM	2	1	40.92	18.69	37.85
CNNDM	2	8	41.02	18.78	37.96
CNNDM	6	8	41.03	18.69	37.86
NYT	2	1	43.64	25.58	36.87
NYT	2	8	44.11	26.08	37.34
NYT	6	8	43.93	25.99	37.15
Table 7: The in-domain performance of the summarizers.
B Full Results on Overall Tree Parsing
We show the overall results of all the six summarization models on constituency/dependency parsing in Table.8, the results of three of them are shown in Table.2, Table.3 and Table.4 in the main paper.
C Results on Sensitivity to Initialization and Summarizer Quality
To explore if the models with different random initialization have consistent performances, we train 5 models with 6 layers and 8 heads on the CNNDM dataset with different initialization, and the results of each layer for constituency/dependency parsing are shown in Table.9. We can find that the results are relatively consistent across different initialization, and additional exploration on the performance of all the heads (Fig.5) show that, with different initialization of the model, there is consistently one head containing most of the discourse information, but the position of that head is not fixed.
We further do the experiments on dependency parsing during training the summarizer, to see how the performance changes as the summarizer become better, and show the max and mean UAS over three datasets for all attention heads in the 'cnndm-6-8' model by the CLE algorithms after training for (0, 1k, 5k, 10k, 20k) steps in Fig.6. We also show the heatmaps of the average UAS across three datasets for all the heads in Fig.7. We can find that as the summarizer is trained for more steps, more discourse information is learned, and it's more concentrated in one head. Interestingly,