言語処理学会 第25回年次大会 発表論文集 (2019年3月)

Improving Evidence Detection using Warrants asExternal Knowledge
Keshav Singh††1Naoya Inoue†,‡Paul Reisert‡

Kentaro Inui†,‡‡Tohoku University
RIKEN Center for Advanced Intelligence Project (AIP){keshav.singh29, naoya-i, inui}@ecei.tohoku.ac.jp
paul.reisert@riken.jp

Introduction

Evidence detection [12] is an important sub-task ofargument mining and finds an application in a wide
range of natural language comprehension tasks like
identifying support and attack relation [3], essay
evaluation [19] and identifying entailment between
sentences[2]. Given a claim and a set of candidate
premises (or evidences), the evidence detection task
identifies the correct link between the given claim
and a candidate premise. Suppose the following example:(1) Claim: Mary is probably pregnant.
Candidate Premise 1: The pregnancy test waspositive.
Candidate Premise 2: Non-taxable items, promotes illegal activity
The task is to identify Candidate Premise 1 as “evidence” and Candidate Premise 2 as “non-evidence”.
For identifying the link between a claim and premise,a warrant (e.g. “Positive test usually indicates one
is pregnant”) plays an important role. In the past
literature related to argumentation, the use of warrants as a way to establish the link between a claim
and premise has been studied extensively [7, 6, 8].
[14, 7] explain the role of warrants as an external evidence, useful in linking the premise and the claim.
The Toulmin model [20] is also widely accepted tobe both well-structured and general, and has beenshown to be useful for identifying relations between
argumentative components in many argumentative
texts. Nevertheless, previous work on evidence detection [12] and argumentative relation identification [3, 13, etc.] does not use warrants as a hint
for prediction. Therefore, it remains an open-ended
question as to whether or not warrants can be used
as external knowledge for improving evidence detection.
This paper explores a new approach for identifying
the link between a claim and premise by using warrants as external knowledge useful for evidence detection. We hypothesize that a set of warrants related tothe claim functions as a hint for claim-premise link
identification. We model a deep architecture thatcaptures the link between a premise and claim to explore the applicability of external knowledge in formof warrants for evidence detection. Our preliminary
experiments demonstrate the effectiveness of using
warrants for the task of evidence detection.

2

Related work

Evidence detection has received wide attention over
the years. [12] devised a method for detecting relevant claims to a given debate topic. [17] created abenchmark dataset for identifying context-dependent
evidence. However, little work has been done for integrating warrants as background knowledge for detecting evidence.
The work similar to ours comes under the domainof argument mining and argument reasoning, including both supervised and unsupervised models to
identify relations between sentence pairs. Previous
argumentative relation identification approaches relied more on features extracted from argument component, e.g., semantic similarity, word pairs, textual
entailment and so on. However incorporating external knowledge to identify such relations is still an untouched segment. [13]explore discourse structure features for argumentative relation identification. Theirwork focuses on context aware argumentative relation mining that uses features extracted from writing topics as well as from adjacent context sentences.
Similar to this [11] focuses on macro-level information(e.g. argumentative flow) constructed using argumentative relations in a document. [19] approach
the task of relation identification by identifying argumentative discourse structures and consider discourse markers as misleading or insufficient to identify argumentative relations, thus introduce theirown feature sets. Also instead of taking sentences asinputs their inputs vary from clauses to multiple sentences. [15] present first data-driven model of argumentation structure where they optimize argumenta-― 1241 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Figure 2: Instance from Argument Reasoning Comprehension task.

Figure 1: Model Architecturetion structure globally for the complete sequence ofinput components. But they do not account for nonrelevant components appearing in the argumentative
text. All the above approaches work towards identifying either (support/attack) or (support, attack ornone) relation between argumentative text.
In contrast to the work mentioned above, the approach in this paper focuses on identifying correct
premise(evidence) from a set of evidence in a supervised way. For this reason we use warrants to link thecorrect pair of claim and evidence. Where warrants
are established to link the support of source towards
target. In addition, the approach tries to incorporate
background knowledge in the form of warrants which
we believe is the first attempt towards this task. Ouridea, correlates to the approach made by [1], where
they try to fill in the gap between claim and premiseby letting annotators write down implicit warrants.
They show that using manually-compiled premises
improves similarity-based claim matching and thatpremises generalize to unseen user claims. But they
concluded only with a preliminary analysis due tolarge variance in the responses. Also [3] model a
deep learning architecture with identifying relationsof attack and support between natural language arguments in text, by classifying pairs of pieces of text
as attack, support or neither attack nor support relations. We combine the above two approaches partially and use warrants as external knowledge so as
to better identify the correct premise as evidence forthe respective claim using warrants.

3Model3.1

Key idea

better together. We propose a warrant aware model
to better understand relations between C and its P
with the use of these knowledge fragments. Also,
since these warrants are not available exclusively,
we assume that given a claim, we propose to link
together warrants relative to that particular claim.
The reason we do this is, since warrant are rule-like
general statements [21] therefore its applicability isjustified over a wide variety of premises given a claim.
And in such way we can better identify the relations
even when given adversarial instances.

3.2

Architecture

We extend Conneau [5]’s architecture, which is originally proposed for classifying a pair of sentences.
Figure 1 depicts our architecture used for binary classification of supporting evidence and non-supporting
evidence detection task L = {Support, Non-relevant}.
Here, (p, c, w) refers to encoded vector representations of (premise, claim, warrant) in Bm . To get
encode sentences into a fixed-size vector, we use abidirectional LSTM [18]. We concatenate the hidden states of forward and backward LSTM whichreads the sentence in opposite directions. Before getting the fixed size vector, we apply max pooling [4]by selecting maximum value over each dimension ofhidden units.
After generating the vector representation of sentences, 3 matching methods are applied to extract
relations between (p, c, w).
1. Concatenation(p, c, w)ofindividual

representation

2. Element-wise product (p ∗ w, p ∗ c, c ∗ w, c ∗ w ∗ p)
3. Absolute element-wise difference (|p − c|, |c −w|, |p − w|)

We assume that warrants are like external knowledge
fragments necessary to link the claim and premise

All the resulting vectors are concatenated and fed
to a softmax classifier, connected by fully connectedneural network. The softmax layer predicts the label∈ L for the relation between C and P .

― 1242 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

Hyperparameter

Value

Model

Accuracy

Dropout
BiLSTM size
Batch size
Embedding size

0.532
32
100

Baseline model+ Warrant

72.71 ±2.1776.74 ±1.53

Table 2: Performance of claim-premise link detection(averaged over 5 runs with different random seeds).

Table 1: Hyperparameters used in our experiments.

4Experiments

4.1

DatasetFor our task, we use Argument Reasoning Comprehension Corpus (ARCC), the annotated data provided by [9] for SemEval task 2018 [10]. This dataset
fits well in our hypothesis of using warrants for claim
and premise evidence detection. An instance of thedataset is shown in Figure2. Although the initialtask concerning ARC dataset is to identify the correct warrant given two choices as shown in Figure
2. We use this dataset by extracting all the correct
choice warrants and then modifying the dataset so
as to fit it in our assumption (see Sec 3.1).
The dataset covers various topics such as technology, politics, general issues, and so on. In total, we
found that there are 169 different claims and thedataset is very diverse, small and involves multiple
domains which makes the task challenging. In total,
we have 1,210 instances as training data and 444 instances as test data. We slightly change this dataset
and collect all correct warrants.
After processing we are left with each instance as
the tuple T of type (P, C, Wc ), where Wc is the correct warrant for the given instance. We later use thisdata for further use.
In accordance with our assumptions, we make anew dataset from the given (P, C, Wc ) instances.
Given the tuples of (P remise,Claim and Correctwarrant), we link together all the Wc that are associated to their respective claims.
Ci −→ [w1 , w2 , w3 ...]i , i ∈ (1, 169)
Next, we randomly choose one warrant (Wci ) forCi from its respective set [w1 , w2 , w3 ...]i and makea new dataset along with (P ). Hence, creating thenew dataset (P, Ci , Wci ), we move to the next step ofgenerating negative instances with randomly choosing premise for the (Ci , Wci ) tuple. We treat the initial 1,210 instances (P, Ci , Wci ) as Positive Dataset
PD and label them {1}. Next, we randomly selectPr for Pr in (Pr , Ci , Wci ) making sure that Pr does
not match with its original C as in PD . This approach makes sure that Pr and Ci forms neither
support nor attack relation between them. We callthese tuples as Negative instances (Pr , Ci , Wci ) andterm this dataset as Negative Dataset ND and label
them {0}. Further we combine these datasets and get
2420 instances of {Support, Non-relevant} relation,labeled as {1,0 } respectively. Later we apply thesame approach to testing dataset and get 888 testing instances. We term this new combined dataset
as Benchmark Bm = PD ∈ (P, Ci , Wci , 1) ∪ ND ∈(Pr , Ci , Wci , 0).

4.2

Settings

Word embeddings are initialized with 100dimensional GloVe vectors [16].
The words
that do not appear in GloVe are initialized with arandom value. We use the same hyperparameters
for training both models. The values of hyperparameters used in our experiments are summarized inTable 1. We employ early stopping with a patienceof 15. Adagrad [22] is used for optimization.
During training, we divide the dataset of ARCC
into a training dataset and validation dataset with
8:2 split. We then get the best weights on the validation dataset and use it in the testing phase. To
make sure that the dataset is evenly distributed inboth the training and validation set, we used a stratified shuffle split. For the testing phase, we use 888
instances from the test dataset of ARCC.
In our experiments, we compare two models: (i)baseline model : our model without the warrant encoding layer, and (ii) proposed model : our modeltrained on Bm dataset along with warrants.

4.3

Results

The results of our experiments are shown in Table 2. We observe that using warrants as backgroundknowledge significantly improves the performance ofour model. We attribute this to the fact that apremise and a claim are more likely to be related(i.e., the premise acts as evidence for the claim) if awarrant links both of them.

― 1243 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.

5Conclusion and Future work

This paper has presented a first approach to incorporate background knowledge in the form of warrants to identify the correct premise from a given
set, for a given claim. The results reveal that having in-domain knowledge in small fragments can help
identify the correct premise significantly. Although
the results look promising but there is still room forfurther improvement for exploring this hypothesis.
The results obtained motivate us to dwell deeper
in the direction of using external knowledge. For
future work, we plan to investigate graphical way ofrepresenting domain specific knowledge and incorporate it to improve evidence detection task. Also, wewill extend our approach to different corpora withgraphical knowledge representation.

Acknowledgements
This work was partially supported by JSPS KAKENHI Grant Number 15H01702 and JST CREST
Grant Number JPMJCR1513, including AIP challenge.

References[1] Filip Boltuzic and Jan Šnajder. Fill the gap! analyzing
implicit premises between claims from online debates. In
Proceedings of the Third Workshop on Argument Mining(ArgMining2016), pages 124–133, 2016.
[2] Samuel R Bowman, Gabor Angeli, Christopher Potts,
and Christopher D Manning. A large annotated corpus
for learning natural language inference. arXiv preprint
arXiv:1508.05326, 2015.
[3] Oana Cocarascu and Francesca Toni. Identifying attackand support argumentative relations using deep learning. In Proceedings of the 2017 Conference on Empirical
Methods in Natural Language Processing, pages 1374–1379, 2017.
[4] Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neuralnetworks with multitask learning. In Proceedings ofthe 25th international conference on Machine learning,pages 160–167. ACM, 2008.

warrants. In Proceedings of the 2018 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies,
Volume 1 (Long Papers), volume 1, pages 1930–1940,
2018.
[10] Ivan Habernal, Henning Wachsmuth, Iryna Gurevych,and Benno Stein. Semeval-2018 task 12: The argument
reasoning comprehension task. In Proceedings of The
12th International Workshop on Semantic Evaluation,pages 763–772, 2018.
[11] Tatsuki Kuribayashi, Paul Reisert, Naoya Inoue, andKentaro Inui. Towards exploiting argumentative context
for argumentative relation identification. 2018.
[12] Marco Lippi and Paolo Torroni. Context-independentclaim detection for argument mining. In IJCAI, volume 15, pages 185–191, 2015.
[13] Huy Nguyen and Diane Litman. Context-aware argumentative relation mining. In Proceedings of the 54th
Annual Meeting of the Association for ComputationalLinguistics (Volume 1: Long Papers), volume 1, pages
1127–1137, 2016.
[14] E Michael Nussbaum. Argumentation, dialogue theory,and probability modeling: Alternative frameworks for argumentation research in education. Educational Psychologist, 46(2):84–106, 2011.
[15] Andreas Peldszus and Manfred Stede. Joint prediction
in mst-style discourse parsing for argumentation mining. In Proceedings of the 2015 Conference on Empirical
Methods in Natural Language Processing, pages 938–948,
2015.
[16] Jeffrey Pennington, Richard Socher, and Christopher
Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empiricalmethods in natural language processing (EMNLP), pages
1532–1543, 2014.
[17] Ruty Rinott, Lena Dankin, Carlos Alzate Perez,
Mitesh M Khapra, Ehud Aharoni, and Noam Slonim.
Show me your evidence-an automatic method for context
dependent evidence detection. In Proceedings of the 2015
Conference on Empirical Methods in Natural Language
Processing, pages 440–450, 2015.
[18] Mike Schuster and Kuldip K Paliwal. Bidirectional recurrent neural networks. IEEE Transactions on Signal
Processing, 45(11):2673–2681, 1997.
[19] Christian Stab and Iryna Gurevych. Identifying argumentative discourse structures in persuasive essays. InProceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages
46–56, 2014.
[20] Stephen E Toulmin. The uses of argument. Cambridge
university press, 2003.

[5] Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364, 2017.

[21] Bart Verheij. Evaluating arguments based on toulmins
scheme. Argumentation, 19(3):347–371, 2005.

[6] James B Freeman. Relevance, warrants, backing, inductive support. Argumentation, 6(2):219–275, 1992.

[22] Matthew D Zeiler. Adadelta: an adaptive learning rate
method. arXiv preprint arXiv:1212.5701, 2012.

[7] James B Freeman. Argument strength, the toulmin
model, and ampliative probability.
Informal Logic,
26(1):25–40, 2008.
[8] G Thomas Goodnight. Legitimation inferences: An additional component for the toulmin model. Informal Logic,
15(1), 1993.
[9] Ivan Habernal, Henning Wachsmuth, Iryna Gurevych,and Benno Stein. The argument reasoning comprehension task: Identification and reconstruction of implicit― 1244 ―Copyright(C) 2019 The Association for Natural Language Processing.
All Rights Reserved.