SlowGuess/NAACL_2022_OCR huggingface dataset:
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7871-7880, Online. Association for Computational Linguistics.
Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014. Text-level discourse dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25-35, Baltimore, Maryland. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
William C Mann and Sandra A Thompson. 1988. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243-281.
David Mareček and Rudolf Rosa. 2019. From balustrades to pierre vinken: Looking for syntax in transformer self-attentions. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 263-275, Florence, Italy. Association for Computational Linguistics.
Julian Michael, Jan A. Botha, and Ian Tenney. 2020. Asking without telling: Exploring latent ontologies in contextual representations. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6792-6812, Online. Association for Computational Linguistics.
Mathieu Morey, Philippe Muller, and Nicholas Asher. 2017. How much progress have we made on RST discourse parsing? a replication study of recent results on the RST-DT. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1319-1324, Copenhagen, Denmark. Association for Computational Linguistics.
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulíçehre, and Bing Xiang. 2016. Abstractive text summarization using sequence-to-sequence RNNs and beyond. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018
Conference on Empirical Methods in Natural Language Processing, pages 1797-1807, Brussels, Belgium. Association for Computational Linguistics.
Bita Nejat, Giuseppe Carenini, and Raymond Ng. 2017. Exploring joint neural model for sentence level discourse parsing and sentiment analysis. In Proceedings of the 18th Annual SIGdial Meeting on Discourse and Dialogue, pages 289-298, Saarbrücken, Germany. Association for Computational Linguistics.
Thanh-Tung Nguyen, Xuan-Phi Nguyen, Shafiq Joty, and Xiaoli Li. 2021. RST parsing from scratch. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1613–1625, Online. Association for Computational Linguistics.
Barlas O'Guz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Wen-tau Yih, Sonal Gupta, et al. 2021. Domain-matched pretraining tasks for dense retrieval. arXiv preprint arXiv:2107.13602.
Lalchand Pandia, Yan Cong, and Allyson Ettinger. 2021. Pragmatic competence of pre-trained language models through the lens of discourse connectives. In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 367-379, Online. Association for Computational Linguistics.
Yannis Papanikolaou, Ian Roberts, and Andrea Pierleoni. 2019. Deep bidirectional transformers for relation extraction without supervision. In Proceedings of the 2nd Workshop on Deep Learning Approaches for Low-Resource NLP (DeepLo 2019), pages 67-75, Hong Kong, China. Association for Computational Linguistics.
Rashmi Prasad, Nikhil Dinesh, Alan Lee, Eleni Miltsakaki, Livio Robaldo, Aravind Joshi, and Bonnie Webber. 2008. The Penn Discourse TreeBank 2.0. In Proceedings of the Sixth International Conference on Language Resources and Evaluation (LREC'08), Marrakech, Morocco. European Language Resources Association (ELRA).
Jack W. Rae, Anna Potapenko, Siddhant M. Jayakumar, Chloe Hillier, and Timothy P. Lillicrap. 2020. Compressive transformers for long-range sequence modelling. In International Conference on Learning Representations.
Alessandro Raganato and Jörg Tiedemann. 2018. An analysis of encoder representations in transformer-based machine translation. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 287-297, Brussels, Belgium. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for
machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky. 2020. A primer in BERTology: What we know about how BERT works. Transactions of the Association for Computational Linguistics, 8:842-866.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184-188, Vancouver, Canada. Association for Computational Linguistics.
Bonnie Webber, Matthew Stone, Aravind Joshi, and Alistair Knott. 2003. Anaphora and discourse structure. Computational Linguistics, 29(4):545-587.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Zhiyong Wu, Yun Chen, Ben Kao, and Qun Liu. 2020. Perturbed masking: Parameter-free probing for analyzing and interpreting BERT. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4166-4176, Online. Association for Computational Linguistics.
Wen Xiao, Iz Beltagy, Giuseppe Carenini, and Arman Cohan. 2021a. Primer: Pyramid-based masked sentence pre-training for multi-document summarization. arXiv preprint arXiv:2110.08499.
Wen Xiao, Patrick Huber, and Giuseppe Carenini. 2021b. Predicting discourse trees from transformer-based neural summarizers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4139-4152, Online. Association for Computational Linguistics.
Linzi Xing, Wen Xiao, and Giuseppe Carenini. 2021. Demoting the lead bias in news summarization via alternating adversarial learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 948-954, Online. Association for Computational Linguistics.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Manzil Zaheer, Guru Prashanth Guruganesh, Avi Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontonon, Philip Minh Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Mahmoud El Houssieny Ahmed. 2020. Big bird: Transformers for longer sequences. In Advances in Neural Information Processing Systems.
Amir Zeldes. 2017. The GUM corpus: Creating multilayer resources in the classroom. *Language Resources and Evaluation*, 51(3):581-612.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pages 11328-11339. PMLR.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, page 649-657, Cambridge, MA, USA. MIT Press.
Zining Zhu, Chuer Pan, Mohamed Abdalla, and Frank Rudzicz. 2020. Examining the rhetorical capacities of neural language models. In Proceedings of the Third Blackbox NLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pages 16-32, Online. Association for Computational Linguistics.
A Huggingface Models
We investigate 7 fine-tuned BERT and BART models from the huggingface model library, as well as the two pre-trained models. The model names and links are provided in Table 3
Pre-Trained	Fine-Tuned	Link
BERT-base	-	https://huggingface.co/bert-base-uncased
BERT-base	IMDB	https://huggingface.co/textattack/bert-base-uncased-imdb
BERT-base	Yelp	https://huggingface.co/fabriceyhco/bert-base-uncased-yelp_polarity
BERT-base	SST-2	https://huggingface.co/textattack/bert-base-uncased-SST-2
BERT-base	MNLI	https://huggingface.co/textattack/bert-base-uncased-MNLI
BART-large	-	https://huggingface.co/facebook/bart-large
BART-large	CNN-DM	https://huggingface.co/facebook/bart-large-cnn
BART-large	XSUM	https://huggingface.co/facebook/bart-large-xsum
BART-large	SQuAD	https://huggingface.co/valhalla/bart-large-finetuned-squadv1
Table 3: Huggingface pre-trained and fine-tuned model links.
B Test-Set Results on RST-DT and GUM
(a) BERT: PLM, +IMDB, +Yelp, +MNLI, +SST-2
(b) BART: PLM, +CNN-DM, +XSUM, +SQuAD
Figure 6: Constituency (top) and dependency (bottom) discourse tree evaluation of BERT (a) and BART (b) models on RST-DT (test). Purple=high score, blue=low score. + indicates fine-tuning dataset.
Figure 7: Constituency (top) and dependency (bottom) discourse tree evaluation of BERT (a) and BART (b) models on GUM (test). Purple=high score, blue=low score. + indicates fine-tuning dataset.
C Oracle-picked self-attention head compared to validation-picked matrix
Model	RST-DT	GUM
Span	UAS	Span	UAS
BERT
rand. init	25.5 (-0.0)	13.3 (-0.0)	23.2 (-0.0)	12.4 (-0.0)
PLM	35.7 (-1.6)	45.3 (-4.9)	33.0 (-0.4)	45.2 (-0.0)
+IMDB	35.4 (-1.8)	42.8 (-2.4)	33.0 (-3.8)	43.3 (-0.1)
+Yelp	34.7 (-1.0)	42.3 (-1.9)	32.6 (-3.6)	43.7 (-0.0)
+SST-2	35.5 (-1.9)	42.9 (-2.5)	32.6 (-0.3)	43.5 (-0.9)
+MNLI	34.8 (-1.7)	41.8 (-1.4)	32.4 (-0.3)	43.3 (-0.5)
BART
rand. init	25.3 (-0.0)	12.5 (-0.0)	23.2 (-0.0)	12.2 (-0.0)
PLM	39.1 (-0.4)	41.7 (-2.7)	31.8 (-0.3)	41.8 (-0.0)
+CNN-DM	40.9 (-0.0)	44.3 (-4.0)	32.7 (-0.3)	42.8 (-0.7)
+XSUM	40.1 (-0.9)	41.9 (-3.4)	32.1 (-1.7)	39.9 (-0.0)
+SQuAD	40.1 (-0.0)	43.2 (-4.6)	31.3 (-2.1)	40.7 (-0.1)
Baselines
Right-Branch/Chain	9.3	40.4	9.4	41.7
Left-Branch/Chain-1	7.5	12.7	1.5	12.2
SumCNN-DM(2021b)	21.4	20.5	17.6	15.8
SumNYT(2021b)	24.0	15.7	18.2	12.6
Two-StagerST-DT(2017)	72.0	71.2	54.0	54.5
Two-StageGUM	65.4	61.7	58.6	56.7
Table 4: Original parseval (Span) and Unlabelled Attachment Score (UAS) of the single best performing oracle self-attention matrix and validation-set picked head (in brackets) of the BERT and BART models compared with baselines and previous work. "rand. init"=Randomly initialized transformer model of similar architecture as the PLM.
D Detailed Self-Attention Statistics
Model	Span	Eisner
Min	Med	Mean	Max	Min	Med	Mean	Max
RST-DT
rand. init	21.7	23.4	23.4	25.5	7.5	10.3	10.3	13.3
PLM	19.3	27.0	27.4	35.7	6.6	17.4	21.6	45.3
+IMDB	19.7	26.9	27.2	35.4	6.6	16.9	21.3	42.8
+YELP	20.2	26.6	26.9	34.7	7.0	16.5	21.0	42.3
+SST-2	19.5	27.3	27.7	35.5	7.3	17.6	21.9	42.9
+MNLI	18.5	26.9	27.1	34.8	6.9	17.5	21.5	41.8
GUM
rand. init	18.6	21.0	21.0	23.2	7.9	10.1	10.1	12.4
PLM	17.8	24.2	24.3	32.6	6.7	16.0	21.2	45.2
+IMDB	18.1	23.8	24.1	32.7	6.1	15.9	21.0	43.3
+YELP	18.6	24.0	23.9	32.3	7.0	15.8	20.7	43.7
+SST-2	18.2	24.6	24.7	32.3	6.5	16.5	21.6	43.5
+MNLI	17.4	23.9	24.2	32.1	6.8	16.6	21.3	43.3
Table 5: Minimum, median, mean and maximum performance of the self-attention matrices on RST-DT and GUM for the BERT model.
Model	Span	Eisner
Min	Med	Mean	Max	Min	Med	Mean	Max
RST-DT
rand. init	20.3	23.3	23.3	25.3	8.5	10.6	10.6	12.5
PLM	20.3	28.3	28.5	39.1	4.1	15.8	19.2	41.7
+ CNN-DM	20.5	28.6	28.7	40.9	3.6	15.2	19.2	44.3
+ XSUM	20.2	27.6	28.3	40.1	4.8	14.8	18.7	41.9
+SQuAD	20.5	27.6	28.2	40.1	2.8	14.8	18.8	43.2
GUM
rand. init	18.6	21.0	21.0	23.2	8.0	10.2	10.2	12.2
PLM	16.7	23.4	23.8	31.5	2.6	15.2	18.7	41.8
+ CNN-DM	15.9	23.7	24.1	32.4	3.7	14.7	18.9	42.8
+ XSUM	16.4	23.2	23.9	31.8	3.0	14.1	18.1	39.9
+SQuAD	16.1	23.4	23.8	31.0	2.4	14.8	18.3	40.7
Table 6: Minimum, median, mean and maximum performance of the self-attention matrices on RST-DT and GUM for the BART model.
E Details of Structural Discourse Similarity
Figure 8: Detailed PLM discourse constituency (left) and dependency (right) structure overlap with baselines and gold trees according to the original parseval and UAS metrics.
Figure 9: Detailed PLM discourse constituency (left) and dependency (right) structure performance of intersection with gold trees (e.g., BERT  $\cap$  Gold Trees  $\leftrightarrow$  Two-Stage (RST-DT)  $\cap$  Gold Trees) according to the original parseval and UAS metrics.
F Intra- and Inter-Model Self-Attention Comparison
Heatmaps sorted by heads (left) and models (right)
(a) BERT constituency tree similarity on GUM
(b) BERT dependency tree similarity on GUM
(c) BART constituency tree similarity on GUM
(d) BART dependency tree similarity on GUM
Figure 10: Top: Visual analysis of sorted heatmaps. Yellow=high score, purple=low score. Bottom: Aggregated similarity of same heads, same models, different heads and different models.  $* =$  Head/  $=$  Model significantly better than  $\neq$  Head/  $\neq$  Model performance with p-value  $< 0.05$ .