SlowGuess/NAACL_2022_OCR huggingface dataset:
Dependency Prediction Table 1 summarizes the performances of different models on both datasets in terms of UAS. For traditional feature engineering models, Two-stage has already achieved satisfactory performance, even beating several neural models like Sent-First LSTM and Complete Parser. This is probably because traditional feature engineering methods design delicate structural features in addition to representations of EDUs so that they can include contextual clues to facilitate parsing. Complete Parser leverages the benefit of better representations from pre-trained transformers to encode the information of individual EDUs, achieving a significant improvement over Sent-First LSTM model with LSTM as primary encoders. However, we show that our model BERT + Sent-First that exploits the potential of Sent-First framework with proper contextualized representations to capture the interactions between individual EDUs and the context surpasses all the existing baselines. The performance of our model can be further improved if we encode contextualized embeddings separately for intra-sentential and intersentential parsing to dynamically capture different information required to parsing at different text granularity levels.
Relation Identification Although previous methods like Graph, Two-stage, and Sent-First LSTM achieve decent results on both datasets, their performances are not comparable to transformer methods developed in recent years. BERT (Cheng21) is our implementation of the state-of-the-art method for relation classification in discourse dependency parsing, which improves the baseline by a large margin. Although BERT is still a very strong baseline in many NLP tasks, direct classification with BERT neglects the contextual clues in the discourse that
Table 2: Model performance of relation identification on SciDTB and CDTB.
Model	SciDTB	CDTB
intra-	inter-	intra-	inter-
Complete Parser (contextualized)	85.6	60.7	79.9	78.0
BERT+Sent-First (shared)	87.6	61.1	81.5	81.6
BERT+Sent-First	88.5	64.7	82.5	82.0
Table 3: Model performance (UAS) on intra- and intersentential dependencies.
can be exploited to aid discourse relation identification, as have been discussed in section 1. We show that the results can be further improved by making use of the sequential structure of the discourse. We design multiple novel sequence labeling models on top of the fine-tuned BERT and all of them achieve a considerable improvement (more than  $1\%$  ) over BERT in terms of accuracy both on the gold dependencies and the predicted dependencies from our Sent-First (separate), showing the benefit of enhancing the interactions between individual EDUs with the context. It yields another large gain when we introduce another layer of inter-sentential level BiLSTM, showing again that it is crucial to capture the interactions between EDUs and their context in both intra- and inter-sentential levels.
5.3 Detailed Analysis
Contextualized Representations for Tree Construction Intuitively, a model should take different views of context when analyzing intra- and inter-sentential structures. As we can see in Table 1, BERT + Sent-First (shared) improves Complete Parser (contextualized) by  $1.2\%$  and  $2.4\%$  on SciTDB and CDTB, respectively. The only difference is BERT + Sent-First makes explicit predictions on two different levels, while Complete Parser (contextualized) treats them equally. When we force BERT + Sent-First to use different BERTs for intra- and inter-sentential analysis, we observe further improvement, around  $3\%$  on both datasets.
If we take a closer look at their performance in intra- and inter-sentential views in Table 3, we can see that BERT + Sent-First (shared) performs better than single BERT model, Complete Parser (contextualized), on both intra- and inter- levels of SciDTB and CDTB, though in some cases we only observe marginal improvement like inter-sentential level of SciDTB. However, when we enhance BERT + Sent-First with different encoders for intra- and inter-sentential analysis, we can observe significant improvement in all cases. That again shows the
BERT	BERT+BiL	BERT+SBiL
intra-	81.8	82.4	82.4
inter-	58.1	60.2	62.6
Table 4: Model performance (classification accuracy) on intra- and inter-sentential relations on SciDTB with gold dependencies. 'ROOT' relation is not counted.
BERT	BERT+BiL	BERT+SBiL
original	72.0	71.8	73.6
modified	50.9	52.3	53.4
Table 5: Model performance (classification accuracy) on automatically generated implicit relation extraction on SciDTB before and after modification.
importance of analyzing with different but more focused contextual representations for the two parsing levels.
Classification or Sequence Labeling? Most previous works treat discourse relation identification as a straightforward classification task, where given two EDUs, a system should identify which relationship the EDU pair hold. As can be seen from Table 2, all sequence labeling models (our main model as well as the variants) achieve a considerable gain over direct classification models on both datasets, especially in terms of accuracy on gold dependencies. This result verifies our hypothesis about the structural patterns of discourse relations shared across different articles. It is noticed that BERT + SBiL performs the best because its hierarchical structure can better capture different structured representations occurring at intra- and inter-sentential levels.
In Table 4, we include the performances of different models on intra- and inter-sentential relations on SciDTB with gold dependency structure. We observe that although our BERT+BiL model improves accuracies on both levels compared to the traditional classification model, the more significant improvement is on the inter-sentential level (by  $2.1\%$ ). We show that it can even be promoted by another  $2.4\%$  if we stack an additional BiLSTM layer on top to explicitly capture the interplay between EDUs on the inter-sentential level. That's probably because writing patterns are more likely to appear in a global view so that discourse relations on the inter-sentential level tend to be more structurally organized than that on the intra-sentential level.
To test the effectiveness of our model for implicit discourse relation identification, We delete some
freely omnissible connectives identified by Ma et al. (2019) to automatically generate implicit discourse relations. This results in 564 implicit instances in the test discourses. We run our model on the modified test data without retraining and compare the accuracies on those generated implicit relations. Table 5 shows the accuracies for those 564 instances before and after the modification. After the modification, although accuracies of all three models drop significantly, our sequence labeling model BERT+BiL and BERT+SBiL outperform the traditional direct classification model BERT by  $1.4\%$  and  $2.5\%$  respectively, showing that our sequence labeling models can make use of clues from the context to help identify relations in the case of implicit relations.
In addition, we experiment with other empirical implementations of contextualized representations instead of averaging tokens like using [CLS] for aggregate representations of sentences for intersentential dependency parsing, but we did not observe a significant difference. Averaging token representations turns out to have better generalizability and more straightforward for implementation.
For the example shown in Figure 1, the relation between EDU 9 and EDU 13 is hard to classify using traditional direct classification because both of them contain only partial information of the sentences but their relation spans across sentences. Therefore, traditional direct classification model gets confused on this EDU pair and predicts the relation to be "elab-addition", which is plausible if we only look at those two EDUs isolated from the context. However, given the gold dependency structure, our sequence labeling model fits the EDU pair into the context and infers from common writing patterns to successfully yield the right prediction "evaluation". This shows that our model can refer to the structural information in the context to help make better predictions of relation labels.
In this paper, we incorporate contextualized representations to our Sent-First general design of the model to dynamically capture different information required for discourse analysis on intra- and intersentential levels. We raise the awareness of taking advantage of writing patterns in discourse parsing and contrive a paradigm shift from direct classifi
cation to sequence labeling for discourse relation identification. We come up with a stacked biLSTM architecture to exploit its hierarchical design to capture structural information occurring at both intra- and inter-sentential levels. Future work will involve making better use of the structural information instead of applying simple sequence labeling.
This work is supported in part by NSFC (62161160339). We would like to thank the anonymous reviewers and action editors for their helpful comments and suggestions.
Stergos Afantenos, Eric Kow, Nicholas Asher, and Jérémy Perret. 2015. Discourse parsing for multi-party chat dialogues. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 928-937, Lisbon, Portugal. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511-521, Baltimore, Maryland. Association for Computational Linguistics.
David Ferrucci, Eric Brown, Jennifer Chu-Carroll, James Fan, David Gondek, Aditya A. Kalyanpur, Adam Lally, J. William Murdock, Eric Nyberg, John Prager, Nico Schlaefer, and Chris Welty. 2010. Building watson: An overview of the deeppa project. AI Magazine, 31(3):59-79.
Shyh-Shiun Hung, Hen-Hsen Huang, and Hsin-Hsi Chen. 2020. A complete shift-reduce Chinese discourse parser with robust dynamic oracle. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 133-138, Online. Association for Computational Linguistics.
Yanyan Jia, Yansong Feng, Yuan Ye, Chao Lv, Chongde Shi, and Dongyan Zhao. 2018a. Improved discourse parsing with two-step neural transition-based model. ACM Trans. Asian Low-Resour. Lang. Inf. Process., 17(2).
Yanyan Jia, Yuan Ye, Yansong Feng, Yuxuan Lai, Rui Yan, and Dongyan Zhao. 2018b. Modeling discourse cohesion for discourse parsing via memory network. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2:
Short Papers), pages 438-443, Melbourne, Australia. Association for Computational Linguistics.
Shafiq Joty, Giuseppe Carenini, and Raymond Ng. 2012. A novel discriminative framework for sentence-level discourse analysis. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, pages 904-915, Jeju Island, Korea. Association for Computational Linguistics.
Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining intra- and multisentential rhetorical parsing for document-level discourse analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 486-496, Sofia, Bulgaria. Association for Computational Linguistics.
Naoki Kobayashi, Tsutomu Hirao, Hidetaka Kamigaito, Manabu Okumura, and Masaaki Nagata. 2020. Top-down rst parsing utilizing granularity levels in documents. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8099-8106.
Sujian Li, Liang Wang, Ziqiang Cao, and Wenjie Li. 2014a. Text-level discourse dependency parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 25-35, Baltimore, Maryland. Association for Computational Linguistics.
Yancui Li, Wenhe Feng, Jing Sun, Fang Kong, and Guodong Zhou. 2014b. Building Chinese discourse corpus with connective-driven dependency tree structure. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2105-2114, Doha, Qatar. Association for Computational Linguistics.
Mingyu Derek Ma, Kevin Bowden, Jiaqi Wu, Wen Cui, and Marilyn Walker. 2019. Implicit discourse relation identification for open-domain dialogues. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 666-672, Florence, Italy. Association for Computational Linguistics.
Joakim Nivre. 2003. An efficient algorithm for projective dependency parsing. In Proceedings of the Eighth International Conference on Parsing Technologies, pages 149-160, Nancy, France.
Jérémy Perret, Stergos Afantenos, Nicholas Asher, and Mathieu Morey. 2016. Integer linear programming for discourse parsing. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 99-109, San Diego, California. Association for Computational Linguistics.
Zhouxing Shi and Minlie Huang. 2018. A deep sequential model for discourse parsing on multi-party dialogues. CoRR, abs/1812.00176.
Ryuichi Takanobu, Minlie Huang, Zhongzhou Zhao, Fenglin Li, Haiqing Chen, Xiaoyan Zhu, and Liqiang Nie. 2018. A weakly supervised method for topic segmentation and labeling in goal-oriented dialogues via reinforcement learning. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4403-4410. International Joint Conferences on Artificial Intelligence Organization.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. CoRR, abs/1706.03762.
Suzan Verberne, Lou Boves, Nelleke Oostdijk, and Peter-Arno Coppen. 2007. Evaluating discourse-based answer extraction for why-question answering. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '07, page 735-736, New York, NY, USA. Association for Computing Machinery.
Yizhong Wang, Sujian Li, and Houfeng Wang. 2017. A two-stage parsing method for text-level discourse analysis. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 184-188, Vancouver, Canada. Association for Computational Linguistics.
An Yang and Sujian Li. 2018. SciDTB: Discourse dependency TreeBank for scientific abstracts. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 444-449, Melbourne, Australia. Association for Computational Linguistics.
Cheng Yi, Li Sujian, and Li Yueyuan. 2021. Unifying discourse resources with dependency framework. In Proceedings of the 20th Chinese National Conference on Computational Linguistics, pages 1058-1065, Huhhot, China. Chinese Information Processing Society of China.
Theorem 1: For a document  $D$  with  $m$  sentences  $(s_1, s_2, \ldots, s_m)$  and  $n$  of the sentences have length (in terms of the number of EDUs) greater or equal to 2 satisfying  $|s_i| \geq 2$ . Let  $T$  be the set of all projective dependency trees obtainable from  $D$ , and let  $T'$  be the set of all projective dependency trees obtainable from  $D$  in a Sent-First fashion. Then the following inequality holds:
$$
| T ^ {\prime} | \leq \frac {2}{n + 1} | T |
$$
Proof of Theorem 1: By the definition of our Sent-First method, trees in  $T'$  satisfy the property that there is exactly one EDU in each sentence
whose head or children lies outside the sentence. It is clear that  $T' \subset T$ . We consider a document  $D$  with  $m$  sentences  $(s_1, s_2, \ldots, s_m)$  and  $n$  of the sentences have length(in terms of the number of EDUs) greater or equal to 2 satisfying  $|s_i| \geq 2$ .
$\forall \sigma^{\prime}\in T^{\prime},\sigma^{\prime}$  is a valid projective dependency tree obtainable from  $D$  in a Sent-First fashion. We define a  $t$  -transformation to a sentence  $s_i,|s_i| > 1$  with its local root of the sentence  $e_{ia}$  not being the root of the document in  $\sigma^{\prime}$  with the following rules:
1. If  $e_{ia}$  has no child outside  $s_i$ ,  $e_{ib}$  is its furthest (in terms of distance to  $e_{ia}$ ) child or one of its furthest children inside  $s_i$ , then delete the edge between  $e_{ia}$  and  $e_{ib}$  and set the head of  $e_{ib}$  to be the head of  $e_{ia}$ .
2. Else if  $e_{ia}$  has at least one child before  $e_{ia}$  inside  $s_i$ , and  $e_{ib}$  is its furthest child before  $e_{ia}$  inside  $s_i$ . Delete the edge between  $e_{ia}$  and  $e_{ib}$ . If  $i > 1$ , set the head of  $e_{ib}$  to be the local root of sentence  $s_{i-1}$ , else  $i = 1$ , set the head of  $e_{ib}$  to be the local root of sentence  $s_{i+1}$ .
3. Else,  $e_{ia}$  has at least one child after  $e_{ia}$  inside  $s_i$ , and  $e_{ib}$  is its furthest child after  $e_{ia}$  inside  $s_i$ . Delete the edge between  $e_{ia}$  and  $e_{ib}$ . If  $i < m$ , set the head of  $e_{ib}$  to be the local root of sentence  $s_{i+1}$ , else  $i = m$ , set the head of  $e_{ib}$  to be the local root of sentence  $s_{m-1}$ .
Suppose  $\sigma_{i}$  is obtained by applying  $t$ -transformation to the sentence  $s_i$ , it is obvious to show that  $\sigma_{i} \in T / T'$ .  $n - 1$  valid  $t$ -transformations can be applied to  $\sigma'$ . A reverse transformation  $t^{-1}$  can be applied to  $\sigma_{i}$  with the following rule: if a sentence has two local roots, change the head of one of the roots to the other root. In this way, at most two possibly valid trees  $\in T'$  can be obtained because we are not sure which one is the original local root of the sentence. Therefore, at most 2 different  $\sigma' \in T'$  can be found to share the same tree structure after a  $t$ -transformation. See Figure 5 for illustration. Therefore,
Figure 4: An illustration of transformation  $t$  for Theorem 1.
$$
| T / T ^ {\prime} | \geq \frac {n - 1}{2} | T ^ {\prime} |
$$
Theorem 1 shows that the search space shrinks with the number of sentences. Therefore, Sent-First approach is especially effective at the reduction of search space so that the parser has a better chance to find the correct result, no matter what kind of parser is used specifically. Since the effectiveness has been proved, this approach can even be confidently generalized to other cases where similar sentence-like boundaries can be identified.
Besides, an even stronger bound regarding the use of Sent-First method can also be proved for constituent parsing.
Theorem 2: For a document  $D$  with  $m > 1$  sentences  $(s_1, s_2, \ldots, s_m)$  and  $n$  of the sentences have length (in terms of the number of EDUs) greater or equal to 2 satisfying  $|s_i| \geq 2$ . Let  $T$  be the set of all binary constituency trees obtainable from  $D$ , and let  $T'$  be the set of all binary constituency trees obtainable from  $D$  in a Sent-First fashion. Then the following inequality holds:
$$
| T ^ {\prime} | \leq (\frac {1}{2}) ^ {n} | T |
$$
Proof of Theorem 2: By the definition of our Sent-First method, trees in  $T'$  satisfy the property that EDUs in a sentence forms a complete subtree. It is clear that  $T' \subset T$ . We define a tree transformation  $t$ , for a tree  $u_1$  with child  $u_2$  and  $u_3$ ,  $u_3$  being a complete discourse tree of a sentence with more than 2 EDUs.  $u_3$  must also have 2 children named  $u_4$  and  $u_5$  where  $u_4$  is adjacent to  $u_2$  in the sentence. After transformation  $t$ , a new tree  $u_1'$  is derived whose children are  $u_5$  and a subtree  $u_6$  with children  $u_2$  and  $u_4$ .  $u_1 \in T'$ , while  $u_1' \in T / T'$ . Illustration see Figure 6. Note that  $t$  is one-to-one so that different  $u_1$  will be transformed to different  $u_1'$  after  $t$ -transformation and  $u_1$  can be applied  $t$ -transformation twice if both children of  $u_1$  are complete DTs for a sentence (more possible trees  $u_1'$  can be transformed into if the order of transformation is also considered). Transformation  $t$  is a local transformation and does not affect sub-trees  $u_2, u_4$ , and  $u_5$ .
$\forall \sigma^{\prime}\in T^{\prime},\sigma^{\prime}$  is a valid projective dependency tree obtainable from  $D$  in a Sent-First fashion. Since all sub-trees representing a sentence must merge into one complete discourse tree representing the whole document, there must be  $n$  inde
Figure 5: An illustration of transformation  $t$  for Theorem 2.
pendent  $t$  transformations applicable to some subtrees in  $\sigma^{\prime}$ , so that at least  $2^{n} - 1$  trees can be obtained after  $i\geq 1$ $t$  transformations  $\in T / T^{\prime}$ . Since  $t$ -transformation is one-to-one,  $\forall \sigma_1,\sigma_2\in T',\sigma_1\neq \sigma_2,\sigma_1'$  is a tree obtained after some  $t$ -transformations on  $\sigma_{1},\sigma_{2}^{\prime}$  is a tree obtained after some  $t$ -transformations on  $\sigma_{2},\sigma_{1}^{\prime}\neq \sigma_{2}^{\prime}$ .
$$
\left| T / T ^ {\prime} \right| \geq (2 ^ {n} - 1) \left| T ^ {\prime} \right|
$$
$$
\left| T ^ {\prime} \right| \leq \left(\frac {1}{2}\right) ^ {n} | T |
$$
B Additional Detailed Results
Relation	BERT	BERT+BiL	BERT+SBiL
elab-addition	77.5	78.9	80.2
evaluation	76.3	77.8	81.6
joint	81.7	80.4	82.5
attribution	92.7	95.5	95.5
enablement	82.1	84.1	83.4
manner-means	86.2	85.0	86.2
contrast	73.9	75.0	77.1
bg-goal	59.3	63.5	67.7
same-unit	89.7	93.2	93.2
progression	19.0	6.1	15.4
bg-compare	43.8	44.1	60.9
elab-aspect	29.2	28.1	36.2
bg-general	70.2	94.3	91.7
condition	57.1	54.2	52.0
Table 6: Model performance (F1 score) for the 14 most frequent relation types on gold dependencies of SciDTB. The first 14 relations are listed in descending order in terms of their frequencies in the test dataset (652, 178, 156, 131, 127, 121, 71, 56, 54, 48, 46, 45, 37, 33).
Span	BERT	BERT+BiL	BERT+SBiL
1	82.7	83.1	82.9
2	63.6	67.5	67.1
3	51.6	55.6	59.5
4	61.0	58.4	59.7
5	52.2	53.7	62.7
6	63.0	63.0	60.9
7	70.6	73.5	58.9
8	52.9	50.0	73.5
9	64.0	64.0	64.0
Table 7: Model performance (accuracy) of relations with gold dependencies on SciDTB against their spans.