Title: Here's My Point: Argumentation Mining with Pointer Networks
Authors: Peter Potash
Alexey Romanov
Anna Rumshisky
Venue: ICLR 2017
Keywords: Natural language processing

Abstract: One of the major goals in automated argumentation mining is to uncover the argument structure present in argumentative text. In order to determine this structure, one must understand how different individual components of the overall argument are linked. General consensus in this field dictates that the argument components form a hierarchy of persuasion, which manifests itself in a tree structure. This work provides the first neural network-based approach to argumentation mining, focusing on extracting links between argument components, with a secondary focus on classifying types of argument components. In order to solve this problem, we propose to use a modification of a Pointer Network architecture. A Pointer Network is appealing for this task for the following reasons: 1) It takes into account the sequential nature of argument components; 2) By construction, it enforces certain properties of the tree structure present in argument relations; 3) The hidden representations can be applied to auxiliary tasks. In order to extend the contribution of the original Pointer Network model, we construct a joint model that simultaneously attempts to learn the type of argument component, as well as continuing to predict links between argument components. The proposed model achieves state-of-the-art results on two separate evaluation corpora. Furthermore, our results show that optimizing for both tasks, as well as adding a fully-connected layer prior to recurrent neural network input, is crucial for high performance.

TL;DR: We use a modified Pointer Network to predict 1) types of argument components; 2) links between argument components.

Review discussion:
Is it the case that every example in your corpora is structured as a tree, or are there examples of other structures as well (such as forests of non-connected trees)?
If it is the case that the data is guaranteed to be tree-structured, why did you only weakly/indirectly enforce this constraint? What tradeoffs do you see between a model like yours and something more directly parsing-inspired that could guarantee a tree-structured output?
Whether or not an example is a single tree depends on the corpus. In the microtext corpus (Peldszus, 2014), all examples are single trees.
In fact, one of the models we compare against, MP+p (Peldszus & Stede, 2015), uses a minimum spanning tree parser.
Alternatively, the persuasive essay corpus (Stab & Gurevych, 2016) includes examples that are not single trees.
In other words, some examples can be forests.
Therefore, we do not want a model that has a single tree assumption. This allows our model to be more generalizable across corpora.
Andreas Peldszus. Towards segment-based recognition of argumentation structure in short texts. ACL 2014, pp. 88, 2014.
Andreas Peldszus and Manfred Stede. Joint prediction in mst-style discourse parsing for argumentation mining. In Proc. of the Conference on Empirical Methods in Natural Language Processing, pp. 938–948, 2015.
Christian Stab and Iryna Gurevych. Parsing argumentation structures in persuasive essays. arXiv preprint arXiv:1604.07370, 2016.
1.	Could the authors explain the reasons they choose to use the pointer network to predict tree structures?
While the proposed model makes sense, there are several other ways of predicting tree structures in the NLP literatures such as using stack-based models or adding dynamic programming layers. Are there any particular advantages of using the proposed structures?
2.	Do the baseline models (classifiers) use word embedding or brown clustering information? It is not clear to me how much of the benefit is from the pre-trained embeddings and how much is from the proposed structures.
Question 1: The PN-based framework easily handles projective and non-projective structures, multi-root parse fragments, as well as components with no incoming or no outgoing links.
In contrast, stack-based models, such as the one proposed by Dyer et al. (2016), typically assume a single-root, projective tree, which is an assumption commonly violated in discourse parsing tasks such as argumentation mining.
Question 2: The model proposed by Stab and Gurevych (2016) uses word embeddings.
Therefore, the performance of our model is not due solely to the presence of embeddings. None of the models, including ours, use Brown Clusters. Please refer to Table 3 for an ablation study of the features used.
Dyer, Chris, et al. "Recurrent neural network grammars." arXiv preprint arXiv:1602.07776 (2016).
Stab, Christian, and Iryna Gurevych. "Parsing Argumentation Structures in Persuasive Essays." arXiv preprint arXiv:1604.07370 (2016).
Among many tasks that PN can handle, how do you characterize the problem, i.e., prediction AC types and links? I think this is an application of PN to some specific problme, but does the paper imply more than that?
Most importantly, we show how to augment the standard PN model to allow for predicting multiple subtasks in this discourse parsing task -- predicting argument relations as well as component types. To reiterate, while a standard PN does sequence modeling, our methodology simultaneously performs classification and sequence modeling.
In fact, as visible in Table 1, the joint modeling is *imperative* for achieving state-of-the-art on this task. Our approach is suitable for other discourse parsing tasks that require either multiple subtasks or joint modeling, such as dialogue parsing (Granell et al., 2009; Tamarit et al., 2012) and temporal relation extraction (Do et al., 2012; Derczynski, 2016).
Our work is the first to use an attention-based model to predict discourse structure -- argumentative structure, in our case. Models such as Yang et al. (2016) and Lin et al. (2016) do incorporate attention over text spans, but they do not use attention to model relations among text spans themselves.
Derczynski, Leon. "Representation and Learning of Temporal Relations." Proceedings of the 26th International Conference on Computational Linguistics. Association for Comptuational Linguistics. 2016.
Do, Quang Xuan, Wei Lu, and Dan Roth. "Joint inference for event timeline construction." Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. Association for Computational Linguistics, 2012.
Granell, Ramon, Stephen Pulman, and Carlos-D. Martínez-Hinarejos. "Simultaneous dialogue act segmentation and labelling using lexical and syntactic features." Proceedings of the SIGDIAL 2009 Conference: The 10th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics, 2009.
Lin, Yankai, et al. "Neural relation extraction with selective attention over instances." Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Vol. 1. 2016.
Tamarit, Vicent, Carlos-D. Martinez-Hinarejos, and Jose-Miguel Benedi. "Estimating the number of segments for improving dialogue act labelling." Natural Language Engineering 18.01 (2012): 1-19.
Yang, Zichao, et al. "Hierarchical attention networks for document classification." Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2016.
This paper proposes a model for the task of argumentation mining (labeling the set of relationships between statements expressed as sentence-sized spans in a short text). The model combines a pointer network component that identifies links between statements and a classifier that predicts the roles of these statements. The resulting model works well: It outperforms strong baselines, even on datasets with fewer than 100 training examples.
I don't see any major technical issues with this paper, and the results are strong. I am concerned, though, that the paper doesn't make a substantial novel contribution to representation learning. It focuses on ways to adapt reasonably mature techniques to a novel NLP problem. I think that one of the ACL conferences would be a better fit for this work.
The choice of a pointer network for this problem seems reasonable, though (as noted by other commenters) the paper does not make any substantial comparison with other possible ways of producing trees. The paper does a solid job at breaking down the results quantitatively, but I would appreciate some examples of model output and some qualitative error analysis.
Detail notes:
- Figure 2 appears to have an error. You report that the decoder produces a distribution over input indices only, but you show an example of the network pointing to an output index in one case.
- I don't think "Wei12" is a name.
Thank you for reviewing our paper. Below we have responded to specific points in the review.
“I am concerned, though, that the paper doesn't make a substantial novel contribution to representation learning.”
Although the problem may appear to be a straightforward application of a PN, the results of Table 1 show that a PN alone does not achieve high performance. We extend the PN through joint modeling, as well as adding a FC-layer for LSTM input. As we note in our discussion, these modifications are crucial for achieving state-of-the-art results. The argument for the joint model is that the representations learned in the encoding phase are able to be shared between the dual tasks.
For example, certain component types are likelier to have incoming/outgoing links, therefore information encoded for type prediction can also be useful for link prediction.
Also, we address the fact that this problem can be solved by a standard recurrent model, as opposed to a sequence-to-sequence model.
However, as our results show, while a BLSTM model can match previous state-of-the-art, the PN model performs measurably better. We argue that the better performance of the PN model is due to the separate representations learned during encoding/decoding. This allows components to be modeled separately as incoming/outgoing components.
“...the paper does not make any substantial comparison with other possible ways of producing trees.”
We would like to reiterate that only in the Microtext corpus are all the examples trees; in the persuasive essay corpus some examples are forests (we will work to clarify this in the paper). With this in mind, for the Microtext corpus, we compare against a model, MP+p, that uses a minimum spanning tree parser. We apologize for not disambiguating the abbreviation ‘MSTParser’.
“Figure 2 appears to have an error. You report that the decoder produces a distribution over input indices only, but you show an example of the network pointing to an output index in one case.”
We chose to construct the figure in this way to illustrate that, by pointing to its own index, the model predicts no outgoing link for this components. By pointing to an input component in the Figure, this implies that the component has an outgoing link. We will clarify this in the paper.
This paper addresses the problem of argument mining, which consists of finding argument types and predicting the relationships between the arguments. The authors proposed a pointer network structure to recover the argument relations. They also propose modifications on pointer network to perform joint training on both type and link prediction tasks.
Overall the model is reasonable, but I am not sure if ICLR is the best venue for this work.
My first concern of the paper is on the novelty of the model. Pointer network has been proposed before. The proposed multi-task learning method is interesting, but the authors only verified it on one task. This makes me feel that maybe the submission is more for a NLP conference rather than ICLR.
The authors stated that the pointer network is less restrictive compared to some of the existing tree predicting method.
However, the datasets seem to only contain single trees or forests, and the stack-based method can be used for forest prediction by adding a virtual root node to each example (as done in the dependency parsing tasks).
Therefore, I think the experiments right now cannot reflect the advantages of pointer network models unfortunately.
My second concern of the paper is on the target task. Given that the authors want to analyze the structures between sentences, is the argumentation mining the best dataset?
For example, authors could verify their model by applying it to the other tasks that require tree structures such as dependency parsing.
As for NLP applications, I found that the assumption that the boundaries of AC are given is a very strong constraint, and could potentially limit the usefulness of the proposed model.
Overall, in terms of ML, I also feel that baseline methods the authors compared to are probably strong for the argument mining task, but not necessary strong enough for the general tree/forest prediction tasks (as there are other tree/forest prediction methods).
In terms of NLP applications, I think the assumption of having AC boundaries is too restrictive, and maybe ICLR is not the best venture for this submission.
Thank you for reviewing our paper. We have responded to specific points in the review below.
“Therefore, I think the experiments right now cannot reflect the advantages of pointer network models unfortunately.”
The ILP Joint model uses 18 different hand-engineered features, often requiring external tools such as dependency parsing and POS tagging. Our model outperforms it with minimal feature extraction. The MP+p model assumes single-tree structure explicitly for links (which is unique to the Microtext corpus). Our model outperforms it for link extraction without any explicit single-tree constraint.
“Pointer network has been proposed before.”
As is evident in Table 1, a direct application of a pointer network does not achieve state-of-the-art. Nor does it accomplish the type prediction task.
In fact, it performs substantially worse than the joint model on link prediction, which does achieve state-of-the-art on both tasks in the persuasive essay corpus, as well as link prediction in the Microtext corpus.
“The proposed multi-task learning method is interesting, but the authors only verified it on one task.”
Although we do focus on a single task, we test the model on two different datasets, each with its own characteristics.
For example, the Microtext corpus has only 100 training examples, and our model is still able to achieve state-of-the-art on link prediction.
Furthermore, the Microtext corpus is much more standardized; all examples are trees and there is exactly one claim in each example.
Therefore, even though we focus on a single task, the datasets we test on have varying characteristics, which highlights the generalizability of the model.
“My second concern of the paper is on the target task.”
One novel aspect of our work is the application of a PN-based model to discourse parsing. Our work is the first to use a sequence-to-sequence, attention-based model for discourse parsing.
“...I found that the assumption that the boundaries of AC are given is a very strong constraint, and could potentially limit the usefulness of the proposed model.”
A model proposed by Stab and Gurevych, the authors of the persuasive essay corpus, achieves near human-level performance in identifying argument components: .867 f1 score compared to .886 f1 score.
Alternatively, for the models they propose, the difference for link prediction is much larger: .751 f1 score compared to .854 f1 score. For type prediction, the gap is also larger: .826 f1 score compared to .868 f1 score.
This paper addresses automated argumentation mining using pointer network.
Although the task and the discussion is interesting, the contribution and the novelty is marginal because this is a single-task application of PN among many potential tasks.
Looking at Table 1, I'm worried it is not clear that our proposed joint model is called 'PN' in the table. This is the model that achieves state-of-the-art.
Conversely, 'PN No Type' is a standard PN applied to this task, which achieves substantially worse results. We try to explain in Section 5 which model is which, but this might not be clear. I understand that as a reviewer, its your job to judge the overall contribution of the paper, but I want it to be clear what the results are: Pointer Network performs poorly compared to our proposed joint model.
We would like to thank the reviewers for their comments, and specifically, reviewers #2 and #3, who acknowledge that the proposed model is interesting and works well, outperforming strong baselines.
To summarize, the main contribution of our work is to propose a novel joint model based on pointer network architecture which achieves state-of-the-art results on argumentation mining task with a large gap.
There are three main concerns raised by the reviewers. The first and the main concern is the novelty of the model.  We believe that in part this concern is due to a misunderstanding that occurred because we mislabeled our proposed joint model as “PN” in the results table (see our discussion below).  We think this led some of the reviewers to believe that the paper merely pointer network model to a specific task.
The second concern is about the overall contribution of the paper to representation learning.  We argue that it is precisely the joint representation learned by our model in the encoding phase, as well as the fact that our model supports separate source and target representations for a given text span, that allows us to substantially outperform standard recurrent models.
The third question raised by the reviewers concerns the meaningful comparison to other methods for recovering relations, such as stack-based models for syntactic parsing.  We believe that this comparison is not appropriate in our case.  As a discourse parsing task, argumentation mining requires the flexibility of recovering relations that are quite distinct from syntactic parsing, in that they allow both projective and non-projective structures, multi-root parse fragments, and components with no incoming or no outgoing links.
We give more specific responses to reviewer comments below.
“Pointer network has been proposed before.”
As is evident in Table 1, a direct application of a pointer network (PN) does not achieve state-of-the-art on link prediction.  Neither is it suitable for AC type prediction.
In fact, a direct application of PN performs substantially worse on link prediction than the joint model, which achieves state-of-the-art on both tasks in the persuasive essay corpus, as well as on link prediction in the microtext corpus.
“I am concerned, though, that the paper doesn't make a substantial novel contribution to representation learning.”
We argue that the better performance of the joint PN model is purely due to the representations of the components learned by the model.
When the representation learned in the encoding phase is shared between the two tasks, the model can factor in the information about the types of argument components that are more likely to be linked.  And indeed, our results shows that the information encoded for type prediction is also useful for link prediction, substantially boosting the performance of the joint model.
Furthermore, we also show that the sequence-to-sequence model we propose does much better than standard recurrent models, e.g. Table 1 shows that a BLSTM model can match previous state-of-the-art, which the joint PN model surpasses substantially.  We argue that the better performance of our model is due to the separate representations learned during encoding/decoding. Effectively, our model allows an argument component to have separate representations in its role as a source or a target of a link.
“The proposed multi-task learning method is interesting, but the authors only verified it on one task.”
Although we do focus on a single task, we test the model on two different datasets, each with its own characteristics.
For example, the Microtext corpus has only 100 training examples, and our model is still able to achieve state-of-the-art on link prediction.
Furthermore, compared to the Persuasive Essay corpus, the Microtext corpus is much more standardized; all examples are trees and there is exactly one claim in each example.
Therefore, even though we focus on a single task, the datasets we test on have varying characteristics, which highlights the generalizability of the model.
“...the stack-based method can be used for forest prediction…”
Compared to stack-based models, the PN-based framework is more flexible.
Specifically, this framework easily handles projective and non-projective structures, multi-root parse fragments, and components with no incoming or no outgoing links.
For example, a stack-based model, such as the one proposed by Dyer et al. (2016), assumes a single-root, projective tree, which is an assumption commonly violated in discourse parsing tasks such as argumentation mining.
“...I found that the assumption that the boundaries of AC are given is a very strong constraint, and could potentially limit the usefulness of the proposed model.”
We would like to point out that previous work on AC boundary detection, specifically, the work by the creators of the persuasive essay corpus (Stab & Gurevich 2016) has already achieved near human-level performance in identifying argument components; this is in stark contrast to the previous models for link and type prediction.
“the experiments right now cannot reflect the advantages of pointer network models.”
We believe this is incorrect.
Specifically, The ILP Joint model uses 18 different hand-engineered features, often requiring external tools such as POS tagging. Our model outperforms it with minimal feature extraction. The MP+p model assumes a single-tree structure explicitly for links (which is unique to the Microtext corpus). Our model outperforms it for link extraction without any explicit single-tree constraint.

Meta-review:
role: PC
data: comment: The paper presents an interesting application of pointer networks to the argumentation mining task, and the reviewers found it generally solid. The reviewers generally agree (and I share their concerns) that the contribution on the machine learning side (i.e. the modification to PNs) is not sufficient to warrant publication at ICLR. Moreover, the task is not as standard and extensively studied in NLP; so strong results on the benchmark should not automatically warrant publication at ICLR (e.g., I would probably treat differently standard benchmarks in syntactic parsing of English or machine translation). If judged as an NLP paper, as pointed out by one of reviewers, the lack of qualitative evaluation / error analysis seems also problematic.

Decision: Reject