SlowGuess/ACL_2020_OCR huggingface dataset:
Influence of the encoder and decoder GAT modules As shown in Figure 4(b), without using structure information, the baseline encoder-decoder (Liu et al., 2019) model gives a development BLEU of 46.83. Adding a GAT module to the encoder as described in Section 4.2 increases the BLEU score to 48.35, demonstrating the usefulness of syntax-aware module. Furthermore, adding a GAT module to the decoder as described in Section 4.3 improves the performance to 49.73, which shows that our skeleton structure model is useful. Finally, a combination of both gives a 50.04 BLEU score.
5.4 Final Results
Table 3 shows the final results on the GMB test dataset. We report performances of the baseline and various tree-structure systems using the exact F1-score by COUNTER in addition to BLEU. The observations are consistent with the development set. Our final model, the joint GAT-enc+dec model,
6823
Model	BLEU	exact F1
Liu et al. (2019)
(baseline)	64.96	77.85
GAT-encoder	66.02	78.22
GAT-decoder	66.69	79.14
GAT-enc+dec	68.14	79.94
Liu et al. (2018)	57.61	68.72
Table 4: Results on the sentence-level dataset.
achieves competitive performance, with a exact F1-score of  $71.65\%$ . Our GAT enhanced models outperform the state-of-the-art model. For the vanilla encoder-decoder model, our GAT-encoder obtains a absolute improvement of  $2.84\%$  exact F1-score, which demonstrates that modeling syntax information is useful. The GAT decoder improves the performance to  $70.81\%$ , giving a  $4.25\%$  promotion, which indicates that the skeleton structure is helpful to DRTS parsing.
As shown in Table 3, Tree-LSTM and GCN based systems also give competitive results to the state-of-the-art baseline model, which again demonstrates the effectiveness of modeling tree structures. GCN achieves better performance than Tree-LSTM by  $1.06\%$ , which can be because the GNN-based model obtains global information during layer stacking, but Tree-LSTM can only capture local structural information. GAT performs better than GCN by  $0.84\%$ , showing that GAT is a competitive choice of GNN. Consistent with observations of BLEU scores, our proposed GAT-enc+dec model shows the best performance on both evaluation metrics.
In addition, we perform experiments on sentence-level datasets as shown in Table 4 as well, following Liu et al. (2019). We use the same setup as the document-level structure-aware model. As shown, both the GAT encoder and decoder can bring better results (i.e.,  $0.37\%$  and  $1.29\%$  by the GAT encoder and decoder, respectively), and their combination can give further improvements (i.e.,  $0.80\%$  over the GAT-decoder) significantly, which are consistent with the findings of the document-level parsing. Finally, the sentence-level performance reaches  $79.94\%$ , a new state-of-the-art score. The results demonstrate that our model is also applicable to sentence-level DRTS parsing.
Interestingly, we find that the BLEU metric is highly indicative of model performance. Based on the observed pair of values on the test results, we
Figure 5: Skeleton-level evaluation F1  $(\%)$  results.
Model	Rel_var	Rel	Unary	Binary
baseline	64.13	34.80	39.21	26.38
GAT-encoder	66.67	36.08	40.98	27.10
GAT-decoder	68.32	36.41	42.34	27.22
GAT-enc+dec	68.97	37.09	43.76	27.74
Table 5: Relation-level evaluation F1 (%) results.
are able to approach the correction between BLEU and COUNTER by a line appropriately, demonstrating a faithful alignment to the COUNTER metric. The observation indicates that the BLEU is also a good metric for the task. Noticeably, one advantage of the BLEU is that the metric calculation is much faster (i.e., only several seconds) than the exact-F1 score, since the latter one consumes at least 24 hours as well as  $100\mathrm{G}+$  memory for the evaluation of the test dataset.
We conduct analysis to examine benefits by the structural-aware model. As the decoding process is decomposed into two steps, we examine the respective gains with respect to the two components, namely skeleton prediction and DRU parsing.
Influence on Skeleton Prediction The bracket scoring metric suggested in Section 5.2 is used to measure the performance of skeleton prediction. Figure 5 shows the F1-scores with respect to node types, which are categorized into three types (Section 2), namely (S)DRS, relation and variable. In addition, the overall performance is reported as well. First, we can see that the (S)DRS nodes can achieve the best performance across the three types, the relation nodes rank the second and the variable type has the worst performance. This indicates the relative difficulty in parsing the three types of nodes. In particular, locating a DRU is relatively simpler as (S)DRS connects with DRU directly,
6824
Figure 6: Discourse representation tree structure examples generated by different models: “ $k_{1}$ : At least 27 wives of Israeli rabbis have signed a letter urging Jewish women to avoid dating Arab men.  $k_{4}$ : The letter warns Jewish women that they will suffer if they date Arab men.”
followed by the coarse-grained discourse relations over the DRUs, while variable nodes are much more difficult since the order matters much (i.e., the subscript number in the variable). Second, the tendencies in terms of different models on the three categories are the same as the overall tendency, where our final model can bring the best skeleton performance, and the baseline shows the worst performance. The observation demonstrates the robustness of our proposed structural-aware model: we can achieve consistently better performances on all the types over the baseline.
Influence on relation tuples inside DRUs Further we analyze the model performance on DRU parsing. A strict matching strategy on the relation tuples inside DRUs is used to measure the performance, as described in Section 5.2. Table 5 shows the performances, where the F1-scores of the overall matching, only relation matching as well as unary and binary relation tuples are reported.4 First, we can find that the overall exact matching F1-score is rather low (below 40). When considering the relation performance ignoring the variables, the final F1-score reaches, with an increase of 31.88, which indicates that variable recognition is extremely difficult. Variables in DURs are similar to the variable nodes in skeleton, however the scale of the inside DRU variables is much larger. We further categorize the relation tuples by their number of variables. The unary tuples (i.e. tuples consist of only one
variable node) can obtain better performance than the binary tuples (i.e. tuples consist of two variable nodes), which is reasonable. In addition, we look into the performance in terms of different models. We can see that all structural-aware models can obtain better performances than the baseline on all settings, demonstrating the effectiveness of our proposed models. In particular, the GAT-decoder demonstrates relatively higher performance compared to GAT-encoder, which is consistent with the results observed in Table 3. As expected, the final joint GAT-enc+dec model obtains a better score than both of individual GAT models.
Case study Figure 6 shows one case study to illustrate the gains of our proposed models over the baseline model, where the detailed differences are highlighted with red color. As shown, the baseline model is already able to obtain a strong results with linguistically-motivated copy strategies, constraint-based inference and so on. However, without structural-aware information, the model is ineffective to handle several implicit long-distance dependencies.
For example, the relation of "That  $(x_{16},p_4)$ " is unable to be recognized by the baseline model, while the models with structural-aware GAT decoder can get it correctly. The major reason is that the structural-aware decoder can transmit the information from  $p_4$  to its parent node, which can facilitate the next-step generation of the parent node.
On the other hand, the syntactic information from the input sentences can help the first-step
4There are no relations containing more than two variables according to the corpus statistics.
6825
skeleton disambiguation. For example, as shown in Figure 6, the models without GAT-encoder can misclassify the relations between  $k_{1}$  and  $k_{4}$ , which is the discourse relation between the input two short sentences. The major reason of the misleading may be possibly due to the word "if" in the second sentence, which is one indicator for the After relation. When the syntactic information is encoded by the GAT encoder, the GAT-enc+dec model can learn the fined-grained dependency reduced by the word "if", and thus is able to obtain the accurate relation of the two sentences (i.e., Conti.)
Discourse parsing is one important topic in the NLP. There are several main types of discourse parsing tasks in the literature, including rhetorical structure theory (RST; MANN and Thompson, 1988) based parsing, centering theory (CT; Grosz et al., 1995; Barzilay and Lapata, 2008) based parsing and DRT based parsing in this study.
Discourse Representation Theory (DRT) based parsing is a relatively classic, yet not fully researched semantic analysis task because of its complexity. Le and Zuidema (2012) present the first work of a data-driven DRT parser, using a graph-based representation of DRT structures. Recently, van Noord et al. (2018b) apply the idea of neural machine translation for graph-based DRT parsing, achieving impressing performance. These studies only focus on sentence-level DRT representations, as the complexity would increase much at the paragraph level. In contrast, we investigate the paragraph level DRT parsing.
DRTS parsing simplifies graphs into trees. There are two existing papers in this line. Liu et al. (2018) are the first to work on DRTS parsing, who propose an end-to-end sequence-to-sequence model for the task. Further, Liu et al. (2019) improve the model by suggesting several effective strategies including supervised attention, copying from alignments, and constraint-based inference. In this work, we improve DRTS parsing instead of Liu et al. (2019) with two types of structure information.
Syntax information has been widely exploited for NLP tasks. Seminal work exploits discrete features designed by experts (Feng and Hirst, 2014; Heilman and Sagae, 2015). Recently, a range of neural modules have been proposed to encode syntax, such as Tree-LSTM (Tai et al., 2015; Zhu et al.; Teng and Zhang, 2016), Tree-CNN (Roy
et al., 2020) and the recently proposed implicit approaches (Yin et al., 2018; Zhang et al., 2019). Syntax has been demonstrated effective for RST based discourse parsing as well (Yu et al., 2018). Our work is to build a syntax tree-aware model and we are the first to use syntax for DRT based discourse parsing.
GNN has received increasing interests for its strong capability of encoding structural information (Kipf and Welling, 2016; Bastings et al., 2017; Zhang et al., 2018; Zhang and Zhang, 2019; Song et al., 2018). GAT is one representative model, which demonstrates success in a number of NLP tasks (Huang and Carley, 2019; Linmei et al., 2019). In this work, we exploit GAT to represent tree-structural information for DRTS parsing.
We investigated the representation of structural information for discourse representation tree structure parsing, showing that a graph neural network can bring significant improvements. In particular, we use GAT for representing syntax in encoding, and representing a structural backbone for decoding. Experiments on the standard GMB dataset show that our method is high effective, achieving the best results in the literature.
We thank all reviewers for the valuable comments, which greatly help to improve the paper. This work is supported by the National Natural Science Foundation of China (NSFC No. 61976180), the funds of Beijing Advanced Innovation Center for Language Resources (No. TYZ19005) and the Westlake University and Bright Dream Joint Institute for Intelligent Robotics. Meishan Zhang is the corresponding author.
Nicholas Asher. 1993. Reference to Abstract Objects in Discourse. Kluwer.
Nicholas Asher and Alex Lascarides. 2003. *Logics of Conversation*. Cambridge University Press.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. CoRR, abs/1409.0473.
Regina Barzilay and Mirella Lapata. 2008. Modeling local coherence: An entity-based approach. Computational Linguistics, 34(1):1-34.
6826
Joost Bastings, Ivan Titov, Wilker Aziz, Diego Marcheggiani, and Khalil Sima'an. 2017. Graph convolutional encoders for syntax-aware neural machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1957-1967, Copenhagen, Denmark. Association for Computational Linguistics.
Johan Bos, Valerio Basile, Kilian Evang, Noortje J. Venhuizen, and Johannes Bjerva. 2017. The Groningen Meaning Bank, pages 463-496. Springer Netherlands, Dordrecht.
Danqi Chen and Christopher Manning. 2014. A fast and accurate dependency parser using neural networks. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 740-750, Doha, Qatar. Association for Computational Linguistics.
Vanessa Wei Feng and Graeme Hirst. 2014. A linear-time bottom-up discourse parser with constraints and post-editing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 511-521, Baltimore, Maryland. Association for Computational Linguistics.
Barbara J. Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics, 21(2):203-225.
Michael Heilman and Kenji Sagae. 2015. Fast rhetorical structure theory discourse parsing. *ArXiv*, abs/1505.02425.
Binxuan Huang and Kathleen Carley. 2019. Syntax-aware aspect level sentiment classification with graph attention networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5468-5476, Hong Kong, China. Association for Computational Linguistics.
Hans Kamp. 1981. A theory of truth and semantic representation. In P. Portner and B. H. Partee, editors, Formal Semantics - the Essential Readings, pages 189-222. Blackwell.
Hans Kamp and Uwe Reyle. 1993. From Discourse to Logic. Dordrecht: Kluwer Academic Publishers.
Jungo Kasai, Dan Friedman, Robert Frank, Dragomir Radev, and Owen Rambow. 2019. Syntax-aware neural semantic role labeling with supertags. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 701-709, Minneapolis, Minnesota. Association for Computational Linguistics.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Phong Le and Willem Zuidema. 2012. Learning compositional semantics for open domain semantic parsing. In Proceedings of COLING 2012, pages 1535-1552, Mumbai, India. The COLING 2012 Organizing Committee.
Jiwei Li, Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1106-1115, Beijing, China. Association for Computational Linguistics.
Zuchao Li, Shexia He, Jiaxun Cai, Zhuosheng Zhang, Hai Zhao, Gongshen Liu, Linlin Li, and Luo Si. 2018. A unified syntax-aware framework for semantic role labeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2401-2411, Brussels, Belgium. Association for Computational Linguistics.
Hu Linmei, Tianchi Yang, Chuan Shi, Houye Ji, and Xiaoli Li. 2019. Heterogeneous graph attention networks for semi-supervised short text classification. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4820-4829, Hong Kong, China. Association for Computational Linguistics.
Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2018. Discourse representation structure parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 429-439, Melbourne, Australia. Association for Computational Linguistics.
Jiangming Liu, Shay B. Cohen, and Mirella Lapata. 2019. Discourse representation parsing for sentences and documents. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6248-6262, Florence, Italy. Association for Computational Linguistics.
WILLIAM MANN and Sandra Thompson. 1988. Rethorical structure theory: Toward a functional theory of text organization. Text, 8:243-281.
Rik van Noord, Lasha Abzianidze, Hessel Haagsma, and Johan Bos. 2018a. Evaluating scoped meaning representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA).
Rik van Noord, Lasha Abzianidze, Antonio Toral, and Johan Bos. 2018b. Exploring neural methods for parsing discourse representation structures. Transactions of the Association for Computational Linguistics, 6:619-633.
6827
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation.
Deboleena Roy, Priyadarshini Panda, and Kaushik Roy. 2020. Tree-cnn: A hierarchical deep convolutional neural network for incremental learning. Neural Networks, 121:148 - 160.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. A graph-to-sequence model for AMR-to-text generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1616-1626, Melbourne, Australia. Association for Computational Linguistics.
Kai Sheng Tai, Richard Socher, and Christopher D. Manning. 2015. Improved semantic representations from tree-structured long short-term memory networks. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1556–1566, Beijing, China. Association for Computational Linguistics.
Zhiyang Teng and Yue Zhang. 2016. Bidirectional tree-structured lstm with head lexicalization.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig. 2018. StructVAE: Tree-structured latent variable models for semi-supervised semantic parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 754–765, Melbourne, Australia. Association for Computational Linguistics.
Nan Yu, Meishan Zhang, and Guohong Fu. 2018. Transition-based neural RST parsing with implicit syntax features. In Proceedings of the 27th International Conference on Computational Linguistics, pages 559-570, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Liwen Zhang, Kewei Tu, and Yue Zhang. 2019. Latent variable sentiment grammar. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4642-4651, Florence, Italy. Association for Computational Linguistics.
Yuan Zhang and Yue Zhang. 2019. Tree communication models for sentiment analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3518-3527, Florence, Italy. Association for Computational Linguistics.
Yue Zhang, Qi Liu, and Linfeng Song. 2018. Sentence-state LSTM for text representation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 317-327, Melbourne, Australia. Association for Computational Linguistics.
Xiaodan Zhu, Parinaz Sobhani, and Hongyu Guo. Long short-term memory over recursive structures. In Proceedings of International Conference on Machine Learning.
6828