SlowGuess/ACL_2020_OCR huggingface dataset:
To illustrate the effectiveness of the latent geometric structure information of argument-relation instances gotten by TransS, we visualize the heat maps of the interaction information of argument representations shown in Figure3. Every word comes with various background colors. The darker patches denote the correlations of word pairs are higher. The example of Comparison relation is listed below:
Arg1: I was prepared to be in a very bad mood tonight.
Arg2: Now, I feel maybe there's a little bit of euphoria.
From the semantics of perspective, this example could be identified as Comparison or Temporal relation. Since argument pairs may have distinct distinguishing features in geometric space, we could consider the geometric structure of argument pairs to help identify the discourse relation. We can obtain the following observations:
- Seen from Figure3(a), without introducing geometric structure information, the model has a high correlation around the word "Now" which might indicate the Temporal relation directly. This demonstrates that only considering the semantic information of arguments may suffer from issues such as polysemy, ambiguity, as well as fuzziness.
- Figure3(b) shows the result of the interaction information of argument representations, which introduces the GSL. From the results, we can see that the model has a high correlation around the word "little" and "very" with the comparative information. The possible reason is that our model utilizing GSL shifts the higher attention from the word "Now" with Temporal information to the word pairs (little, very), (euphoria, bad) and (euphoria, mood) with Comparison relation. Our model with GSL introduces the geometric structure information and jointly utilizes these features and semantic information to help identify the discourse relation.
3.4.4 Impact of Encoder Layer Number
In order to illustrate the impact of the encoder layer number, we select different sizes of encoder layer
Figure 4: The effect of encoder layers' number.
as comparison experiments on the 4-way classification. Figure 4 shows that the  $F_{1}$  scores are increasing until three encoder layers. And when the size of the encoder layer is four or five, the performance of our model is decreasing obviously.
With the increasing of the number of encoder layers, the model could capture the richer semantic information. However, the results imply that with the more encoder layers considered, the model could incur the over-fitting problem due to adding more parameters. Therefore, we adopt three encoder layers to encode the arguments as our Baseline in section 3.3.
Neural network-based models have shown great effectiveness in implicit discourse relation recognition. We give the analysis of mainly relevant work:
4.1 Discourse Argument Representation
Proper argument representation is a core factor of our task. Most previous researches encode arguments as dense and continuous representation based on various neural networks, from basic neural networks (such as CNN, RNN) to complex neural networks (Zhang et al., 2015; Qin et al., 2016b; Rutherford et al., 2016). Some studies adopt different attention or memory mechanisms to catch the emphasis on discourse arguments (Mnih et al., 2014; Liu and Li, 2016; Zhang et al., 2016). Li et al. (2016) exploit the hierarchical attention to capture the focus of different granularities. Zhang et al. (2016) build upon a semantic memory to store knowledge in the distributed fashion for the task. However, these models have only considered the two arguments independently without the interaction information.
4.2 Argument Pair Interactions
Further studies tend to discover more semantic interactions between two arguments by complex neural networks (Qin et al., 2016c; Cai and Zhao, 2017; Lan et al., 2017; Guo et al., 2018). Chen et al. (2016) develop a novel gated relevance network to capture semantic interactions between arguments. Lei et al. (2017) conduct word pair interaction score to capture both linear and quadratic relation for argument representation. However, these methods utilize the pre-trained embeddings for mining the interaction features and ignore the geometric structure information entailed in discourse arguments and their relation.
4.3 Joint Learning Perspective
Recently, some researches adopt joint learning framework to capture more discourse clues for the task. Bai and Zhao (2018) jointly predict connectives and relations, assuming the shared parameters of the deep learning models. Xu et al. (2019) propose a topic tensor network (TTN) to model the sentence-level interactions and topic-level relevance among arguments for this task. However, few studies model discourse relations by translating them in the low-dimensional embedding space as we do in this work.
TransE effectively maps the relation to the embedding space of entities by performing the algebraic operation. Bordes et al. (2013) model entity relations by interpreting them as translating operation in the low-dimensional embedding of the entities. Inspired by TransE, we design a TransS method to mine the latent geometric structure information, which could enhance the argument representations for promoting discourse relation recognition. To our knowledge, this is the first attempt to mine the latent geometric structure of argument-relation. Meanwhile, the embeddings of argument and relation by TransS could be used to the other high-level NLP tasks.
In this paper, we propose a novel TransS-driven joint learning neural network framework by optimizing the discourse argument representations to improve implicit discourse relation recognition. We interpret the discourse relations as translation in low-dimensional embedding space, which reflects the geometric structure of argument-relation, and also can obtain the richer argument representations
based on the multi-level encoder. Different from the conventional approaches only considering the semantic features, we jointly leverage the latent geometric structure information and the semantic features to optimize the argument representations, which could improve the semantic understanding of discourse. Experimental results on the PDTB show the effectiveness of our model.
We thank the anonymous reviewers for their valuable feedback. Our work is supported by the National Key R&D Program of China (2019YFC1521200), the National Natural Science Foundation of China (61976154,U1736103), the Tianjin Natural Science Foundation (18JCYBJC15500), and the Foundation of State Key Laboratory of Cognitive Intelligence, iFLYTEK (CIOS-20190001).
Hongxiao Bai and Hai Zhao. 2018. Deep enhanced representation for implicit discourse relation recognition. In Proceedings of the 27th COLING, pages 571-583.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Proceedings of the 26th NIPS, pages 2787-2795.
Chloe Braud and Pascal Denis. 2015. Comparing word representations for implicit discourse relation classification. In Proceedings of the 2015 EMNLP, pages 2201-2211.
Deng Cai and Hai Zhao. 2017. Pair-aware neural sentence modeling for implicit discourse relation classification. In International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, pages 458-466. Springer.
Jifan Chen, Qi Zhang, Pengfei Liu, Xipeng Qiu, and Xuanjing Huang. 2016. Implicit discourse relation detection via a deep architecture with gated relevance network. In Proceedings of the 54th ACL, pages 1726-1735.
Zeyu Dai and Ruihong Huang. 2018. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. In Proceedings of the 2018 NAACL, pages 141-151.
Shima Gerani, Yashar Mehdad, Giuseppe Carenini, Raymond T. Ng, and Bita Nejat. 2014. Abstractive summarization of product reviews using discourse structure. In Proceedings of the 2014 EMNLP, pages 1602-1613.
Fengyu Guo, Ruifang He, Di Jin, Jianwu Dang, Longbiao Wang, and Xiangang Li. 2018. Implicit discourse relation recognition using neural tensor network with interactive attention and sparse learning. In Proceedings of the 27th COLING, pages 547-558.
Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributional semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329-344.
Man Lan, Jianxiang Wang, Yuanbin Wu, Zheng-Yu Niu, and Haifeng Wang. 2017. Multi-task attention-based neural networks for implicit discourse relationship representation and identification. In Proceedings of the 2017 EMNLP, pages 1299-1308.
Wenqiang Lei, Xuancong Wang, Meichun Liu, Iija Ilievski, Xiangnan He, and Min-Yen Kan. 2017. Swim: A simple word interaction model for implicit discourse relation recognition. In Proceedings of the 26th IJCAI, pages 4026-4032.
Qi Li, Tianshi Li, and Baobao Chang. 2016. Discourse parsing with attention-based hierarchical neural networks. In Proceedings of the 2016 EMNLP, pages 362-371.
Yang Liu and Sujian Li. 2016. Recognizing implicit discourse relations via repeated reading: Neural networks with multi-level attention. In Proceedings of the 2016 EMNLP, pages 1224-1233.
Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In Proceedings of the 30th AAAI, pages 2750-2756.
Thomas Meyer and Andrei Popescu-Belis. 2012. Using sense-labeled discourse connectives for statistical machine translation. In Proceedings of the 13th EACL, pages 129-138.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et al. 2014. Recurrent models of visual attention. In Proceedings of the 27th NIPS, pages 2204-2212.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 NAACL, pages 2227-2237.
Emily Pitler, Annie Louis, and Ani and Nenkova. 2009. Automatic sense prediction for implicit discourse relations in text. In Proceedings of the AFNLP, pages 683-691.
Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016a. Implicit discourse relation recognition with context-aware character-enhanced embeddings. In Proceedings of the 26th COLING, pages 1914-1924.
Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016b. Shallow discourse parsing using convolutional neural network. In CoNLL Shared Task, pages 70-77.
Lianhui Qin, Zhisong Zhang, and Hai Zhao. 2016c. A stacking gated neural architecture for implicit discourse relation classification. In Proceedings of the 2016 EMNLP, pages 2263-2270.
Lianhui Qin, Zhisong Zhang, Hai Zhao, Zhiting Hu, and Eric P. Xing. 2017. Adversarial connective-exploiting networks for implicit discourse relation classification. In Proceedings of the 55th ACL, pages 1006-1017.
Attapol Rutherford and Nianwen Xue. 2014. Discovering implicit discourse relations through brown cluster pair representation and coreference patterns. In Proceedings of the 14th EACL, pages 645-654.
Attapol T Rutherford, Vera Demberg, and Nianwen Xue. 2016. Neural network models for implicit discourse relation classification in english and chinese without surface features. arXiv preprint arXiv:1606.01990.
Sheng Xu, Peifeng Li, Fang Kong, Qiaoming Zhu, and Guodong Zhou. 2019. Topic tensor network for implicit discourse relation recognition in chinese. In Proceedings of the 57th ACL, pages 608-618.
Biao Zhang, Jinsong Su, Deyi Xiong, Yaojie Lu, Hong Duan, and Junfeng Yao. 2015. Shallow convolutional neural network for implicit discourse relation recognition. In Proceedings of the 2015 EMNLP, pages 2230-2235.
Biao Zhang, Deyi Xiong, and Jinsong Su. 2016. Neural discourse relation recognition with semantic memory. arXiv preprint arXiv:1603.03873.