Using clustering techniques to identify
arguments in legal documents
Prakash Poudyal, Teresa Gonçalves, and Paulo Quaresma
Department of Informatics, University of Evora
Evora, Portugal
prakashpoudyal@gmail.com, tcg@uevora.pt, pq@uevora.pt

Abstract. A proposal to automatically identify arguments in legal documents is presented. In this approach, cluster algorithms are applied toargumentative sentences in order to identify arguments. One potential
problem with this process is that an argumentative sentence belonging to
one specific argument can also simultaneously be part of another, distinct
argument. To address this issue, a Fuzzy c-means (FCM) clustering algorithm was used and the proposed approach was evaluated with a set ofcase-law decisions from the European Court of Human Rights (ECHR).
An extensive evaluation of the most relevant and discriminant features
to this task was performed and the obtained results are presented.
In the context of this work two additional algorithms were developed: 1)
the “Distribution of Sentence to the Cluster Algorithm” (DSCA) was developed to transfer fuzzy membership values (between 0 and 1) generated
by the FCM to a set of clusters; 2) the “Appropriate Cluster Identification Algorithm” (ACIA) to evaluate the proposed clusters against thegold-standard clusters defined by human experts.
The overall results are quite promising and may be the basis for further
research work and extensions.
Keywords: Fuzzy clustering algorithm, argument mining

1

Introduction

Advances in communication technology, accessibility of media devices and mushrooming of social media has caused the number of individuals expressing opinions
to grow exponentially. As a result, massive amount of electronic documents are
generated daily, including news editorials, discussion forums and judicial decisions containing legal arguments. In turn, rapid development of current research
into argument mining is raising new challenges for natural language processing
in various fields.
In general to automatically identify a legal argument within an unstructured
text, three stages or modules are used in current practice. The first stage is to
identify the argumentative and non-argumentative sentences, the second stage
is to identify the boundaries of arguments and the third stage is to distinguishthe argument’s components (premise and conclusion).

Using clustering techniques to identify arguments in legal documents

Fig. 1. Overview of the Architecture

To date, second stage processing has been performed by identifying theboundaries of arguments, an extensively explored method in the AI & Law literature. In this paper, we propose a clustering technique that groups argumentative
sentences into a cluster of potential arguments with associated probabilities. An
overview of our approach to the task is shown in Figure 1. The task is complex,
since components of one argument (premise or conclusion) can also be involvedin other arguments.
In the example shown in figure 1 there are 2 distinct arguments. In theexample sentence 2 belongs to argument A and also to Argument B. To clustersuch sentences, we propose to use a Fuzzy c-means (FCM) clustering algorithm[3] that provides a membership value ranging from 0 to 1 for each cluster ofthe sentence. These membership values are key assets of the FCM, since they
allow us to associate each sentence with more than one cluster/argument. The
performance of the FCM depends heavily on the selection of features that areused. In the context of our work we focused mainly on four kinds of features:
N-gram, word2vec, sentence closeness and ‘combined features’. Our aim is to
identify the best performing set of features and techniques to cluster components
to form an argument.
After extracting the features associated with each text, the FCM is used toobtain a cluster membership value for every sentence. To determine the composition of each cluster, we developed a specific algorithm: the “Distribution of
Sentence to the Cluster Algorithm” (DSCA).
To evaluate the performance of the system, a second algorithm, the “Appropriate Cluster Identification Algorithm” (ACIA) was also developed to map
each cluster of the system’s output to the closest matching cluster in the goldstandard dataset.
The rest of the paper is organized as follows: section 3 contains a brief introduction to the datasets. Section 4 deals with measures used for evaluating theperformance of the system. In Section 5 we describe the proposed architecture
including a description of features, a discussion on determining the optimum
number of clusters, and the newly developed DSCA and ACIA algorithms. Section 6 evaluates the performance of all of our experiments. Lastly, Section 7
addresses conclusions and prospects for future work.

Using clustering techniques to identify arguments in legal documents

2State of the Art

In the argument mining field, there has been very limited research about using
clustering techniques to identify and group argumentative sentences into arguments. However, similar problems have been solved by other researchers throughthe use of boundary detection techniques e.g. [14] looked at the places that signal where an argument begins and ends. Since components of an argument canbe spread all over a text, these authors proposed the use of semantic distanceto handle this problem via context-free grammars (CFG). This approach wasapplied to a strictly limited subset of case-law obtaining 60% accuracy. [22] proposed a novel annotation scheme to model arguments. Their approach was to
identify the relation (i.e. ‘support’ or ‘attack’) between the components of arguments. Their technique indicates which premises belong to a claim and constitutethe structure of arguments. [11] performed a manual analysis as well as an automated analysis to detect the boundaries of an argument. To train and test
for automatic analysis, the authors relied on help from experts to analyze thetext manually. For the automatic analysis, they used two Naive Bayes classifiers;
one to identify the first word of a proposition and the other to identify the lastword. [9], [20], and [17] continued this boundary approach, using the Conditional
Random Fields (CRF) algorithm to segment the argument’s components.
[23] is a survey paper about the use of machine learning technologies for argument mining. The paper analyses several approaches made by different authors
regarding argument boundary detection. This review article emphasizes that theboundary detection problems depend upon the adopted argument models. The
main reason is the existence of different types of argument component structure.
Conrad [7] applied a k-means clustering algorithm over plaintiff claims in‘Premises Liability’, ‘Medical Malpractice’ and ‘Racial Discrimination’ suits. The
authors applied their technique to distinguish more effective plaintiff claims fromless effective ones using an ‘award quotient’ metric to segregate the claims. Besides award quotient, the authors used features to help differentiate one cluster’s
properties from another. The authors mention that they also tried aggregative,
partial and graphical features, but didn’t find anything that yielded a performance superior to k-means.

3

The Data Set

We selected case-law documents from the European Court of Human Rights(ECHR)1 annotated by R. Mochales [15]. The corpus is composed of 20 Decision and 22 Judgment categories released before 20 October 1999 by the European Commission on Human Rights. Both categories include similar information, however, the ‘Decision’ present the information briefly with an averageword length of 3,500 words, whereas for ‘Judgment’ the average word length is
10,000 words. We have 9257 sentences, out of which 7097 (77%) of them are1

http://hudoc.echr.coe.int/sites/eng

Using clustering techniques to identify arguments in legal documents

Fig. 2. Proposed Architecture of the Clustering System.

non-argumentative and 2160 (23%) argumentative sentences. Details about the
ECHR corpus is available in [15].

4Evaluation Setup

We used Precision, Recall, F-measure [2, 19] in the analysis of our clustering
technique. Furthermore, we also used cluster purity to evaluate the quality of theobtained clusters. We computed the cluster purity [21] by counting the numberof correctly assigned entities and dividing by the total number of N . Formally
ClusterP urity (ϕ, C) =1 X
maxe=1..k |wd ∩ ce |d=1..kN(1)where N is the summation of the total number of elements in all clusters,
ϕ = {w1 , w1 , · · · wk } is the set of clusters and c = {c1 , c1 , · · · ck } is the setof classes. We interpret wd as the set of sentences in wd and ce as the set ofsentences in ce in Equation 1.

5

System Architecture

Our proposal is to cluster argumentative sentences and, thereby, to identify legal
arguments. Figure 2 presents the system architecture. There are several phases:
first: feature extraction; second: identifying the optimum number of clusters in
the respective case-law file; third: converting the soft clustering values generated
by FCM to hard clustering; and fourth (and final stage): system evaluation.

Using clustering techniques to identify arguments in legal documents

5.1

Feature Extraction

Typically features are values that represent a sentence and are suitable for a machine learning algorithm to handle. It is essential to select the most appropriate
and precise features to train a machine learning algorithm so that the model canbe successfully applied to new data. Therefore, good discriminant features areneeded to correlate similarities between sentences and also address the sequential nature of sentences (since the majority of the components of an argument
are presented in order). To address this requirement, the following features wereused: N-gram [5], word2vec, and sentence closeness (discussed below). Anotherfeature set can also be obtained by combining these three existing features into
what we called “Combined Features”. Each kind of features is discussed below.
Word2vec: The word2vec approach was proposed by [13] and can be implemented in two different ways: as a ‘Continuous Bag of Words’ (CBW) or as a
‘Skip gram’. With Skip-grams, context words are predicted from selected words
in the text, whereas with CBW, a word vector is predicted from the context
of adjacent words. A Wikipedia dump of 05-02-2016 was used as input to theword2vec implementation of Gensim [18], where 100 dimension vectors were generated for each word. From the training set, each word of the sentence is looked
up and its corresponding vector found among the generated word vectors. Then,the average of all vectors of the words presented in the sentence is taken andconsidered to be the ‘sentence vector’.
Sentence Closeness: Sentence closeness is the reciprocal of the inter-sentence
distance (i.e. the distance between sentences) counted in units of whole sentences.
To capture the sequential nature of sentences, distance is a useful feature thathelps to determine which sentences belong to which argument. The highest scoring sentence is considered to be the origin sentence (with a score of 1) fromwhich all other distances are measured. With the exception of the origin sentence, ‘closeness’ scores should decrease monotonically as they move away fromthe origin. Furthermore, as meaning and concepts flow from one sentence to another, this implies that sentences whose ‘closeness’ is high are good candidates
for being clustered together i.e. they belong to the same argument. Equation 2
was used to calculate the ‘closeness’ for each sentence.
1(2)nwhere n the distance between sentences, measured in integer units of sentences.
Combined Features: The previously presented features (N-gram+‘Sentence
Closeness’+Word2vec) were combined into a new feature in an attempt to improve the performance of the clustering algorithms.
Closeness(n) =5.2

Identification of the optimum number of clusters

An argument cluster is a set of sentences which together comprise a single,
coherent legal argument. The process by which sentences are aggregated into
arguments in this way is called clustering. To cluster sentences successfully into

Using clustering techniques to identify arguments in legal documents

Fig. 3. Argument numbers of gold-standard vs. System Prediction (proposed by Xie
and Beni and Cao et al.)arguments, it is currently necessary to specify in advance how many clusters
to expect within a corpus and until recently, there has been no well-establishedapproach to defining this. Techniques that claim to be able to define the optimum
number of clusters in the FCM have been proposed by [24] and Latent Dirichlet
Allocation (LDA) [4].
Employing the Xie and Beni approach [24], we determined experimentallythat an FCM with a fuzziness value of m set to 1.3 in concert with the features
Word2Vec, N-gram and Sentence Closeness, yielded the best results with ourparticular set of case-law files. [24] technique selects the best candidate for thenumber of clusters after obtaining the minimum index value from that respective
cluster number.
The Latent Dirichlet allocation (LDA) technique estimates the number of‘topics’ existing within a text, which means estimating the probabilities of groupings within the text. Inspired by the concept, it was decided to look for suchgroups within our own corpora, and to use the estimated number of topics as
a proxy for the number of clusters. Unfortunately, the LDA technique does not
accept the Word2Vec, TFIDF, and Sentence Closeness features, and so, ‘word
frequency’ in the document was used instead. We selected the ‘CaoJuan2009’
method as a metric which is the best LDA model based on density [16]. ‘CaoJuan2009’ was tested and agreed well with the number of topics (equivalent toour ‘number of clusters’) predicted by the minimum index value [6].
Figure 3 illustrates the results for each experiment: the first is the goldstandard, the second is using Xie and Beni’s proposal, and the third is from
LDA [6]. In the case of Xie and Beni, it can be observed that case-law files 02,
31, 32, 39, 42 find the closest number of clusters to the gold-standard, whereas
for other case-law files the differences are greater.
In case of Cao et al.’s prediction: Case-law files 40 and 41 finds the correct
data required for identification, whereas other case-law files present a slightUsing clustering techniques to identify arguments in legal documents

difference, but not as big as that observed by Xie and Beni. The exact accuracy
score achieved was an identical 8% for both LDA and Xie and Beni.
We also used equation 3, to calculate the difference between the number ofclusters of the gold-standard (Cg ) and the ones predicted by our system (Cs ).
If they differ only by one unit then we considered the prediction is ”almost
correct”; otherwise it’s an incorrect prediction. The result of applying this filter
shows that the accuracy of the closeness scores increase in value to 58% for LDA
and 42% for Xie and Beni, respectively.
|Cs − Cg | ≤ 2(3)

where Cs is the cluster number given by the prediction system, and Cg is thecluster number of the gold-standard.
From the analysis, it’s possible to conclude that LDA achieves a greater accuracy than Xie and Beni and is much closer to the gold-standard. Improvementsin the results are expected to be achieved in the future as more discriminantfeatures are used.

5.3

Clustering Algorithm

After extracting the features, we used a standard Fuzzy c-means (FCM) Clustering algorithm [3] to generate membership values ranging from 0 to 1 for eachcluster. The number of clusters was defined based on the algorithm proposedand described in section 5.2. We set the fuzziness value m ∈ {1.1, 1.3, 2.0}.

5.4

Distribution of Sentence to the Cluster Algorithm (DSCA)

The Distribution of Sentence to the Cluster Algorithm (DSCA) algorithm aims
to transform the membership value generated by FCM (between 0 and 1) into
the set of clusters, each with a membership probability indicating how likely
it is that the sentence belongs to a particular cluster. DSCA is presented as
Algorithm 1
Membership values are represented by a matrix where each row represents a
sentence and each column is labelled with a cluster number (Ci ) ranging from
1 to C. To assign a sentence to the respective cluster, a threshold value t needs
to be specified to help define boundaries between the clusters. The value isdefined only if the difference between the maximum membership value of the ith
position is less than the threshold value for the cluster, otherwise the sentenceis rejected. The algorithm ends after conducting an iterative process through all
positions in the matrix. The concept of threshold value is discussed by [1] as wellas [10]. [1] used a threshold value to eliminate the data points. The authors also
claim that the definition of the appropriate threshold value was determined byexperimentation. After applying the DSCA algorithm, we were able to obtain aproposal for legal arguments: the identified clusters and their sentences.

Using clustering techniques to identify arguments in legal documents

Algorithm 1: Distribution of Sentence to the Cluster Algorithm (DSCA)
1. Denote the matrix of the sentences x cluster by (aij ) ∈ [0, 1], i = 1, 2, 3 · · · S
and j = 1, 2, 3 · · · C such that i stands for sentence and j stands for cluster.
2. Pre-selected threshold (t) is defined
3. for each i do do
imax = max(aij ) ∀ifor each j do do
if (imax − aij ) < t thenselect sentence i for cluster jelse
reject i;
end
end
end
end

6

Result and Evaluation

In order to perform an evaluation of the performance of our system, we needed
to find the best mapping between our system’s clusters and the existing goldstandard data clusters from the ECHR. Therefore, we proposed and developed
a new algorithm the “Appropriate Cluster Identification Algorithm ”(ACIA) tosolve this problem. This algorithm maps the argument predicted by our system
to the closest matching argument in the gold-standard corpus. Here, we describe
the details of the algorithm.
6.1

Appropriate Cluster Identification Algorithm (ACIA)

The ACIA algorithm aims to find the best mapping between the system’s predicted clusters and the gold-standard dataset clusters. A formal description of
the ACIA algorithm is presented in Appendix 8. Here, the steps of the algorithm
are summarized.
Step 1: Initialize the cost value C = 0.
Step 2: Calculate the f1 values between clusters in the gold-standard datasetsand clusters in the system predicted datasets and arranged in matrix form.
Step 3: Select the maximum value from matrix. After value selection, thecorresponding row and column containing the value are removed from future
consideration.
Step 4: The selected value is considered as a node and is inserted with thecost value C = 0 into tree structure.
Step 5: Go back to Step 3 until there are no more eligible f1 values in thematrix left to consider.
Step 6: The total cost of each route in the tree is then calculated. The routewith the maximum cost is then selected.

Using clustering techniques to identify arguments in legal documents

Step 7: These routes consist of f1 values for the best mapping between thesystem predicted clusters and the gold-standard sets.
After identifying the best mapping between the system clusters and the gold
standard arguments, the f1 measure is calculated for each cluster and the overall
average f-measure value is obtained.

Fig. 4. F1 score before and after applying ACIA

In figure 4 we can see the relevance of performing a optimized mapping
between the system’s predicted clusters and the gold-data arguments. We canobserve that the value of ‘After ACIA’ (square symbol) is higher (above 0.3 for
all files), whereas in the case of ‘Before ACIA’ (diamond symbol), the maximum
value never exceeds is 0.3.
6.2

Performance Measurement

The experiment was conducted with the features mentioned in section 5.1 withthe parameters of fuzziness parameter m ∈ {1.1, 1.3, 2.0} and threshold valuet ∈ {0.0001, 0.00001, 0.000001} used for the conversion from a soft to a hardclustering. For the reason of space, we present the results for the features andparameters that score the highest f1 value in most of the case-law files. Table 1
presents the performance result (precision, recall and f1 , cluster purity) showing the N-gram, Sentence closeness, Word2vec and ‘Combine features’ using athreshold value t = 0.00001 and FCM fuzziness (m) = 1.3. Along with this, we
include the number of sentences of each case-law file. The highest f1 value ofeach case-law file obtained from each feature is highlighted in bold and underlined. Case-law files 03, 13, 16, 31, 32 and 42 obtained the highest value using
Word2vec features. Case-law file 02 scored the highest f1 value with N-gram, andcase-law files 30, 35 and 41, the highest f1 with the combined features. Likewise,case-law file 40 scored the highest f1 value with the Sentence Closeness feature.
From this analysis, we can conclude that Word2vec seems to be the best overall
approach.
In comparison to Word2vec, N-gram did not perform as well. The main reason
for this effect is that N-gram uses a bag of words approach which is not effective in

Using clustering techniques to identify arguments in legal documents
Case #S
02
03
13
16
30
31
32
35
39
40
41
42

N-gramSentence ClosenessWord2vec

Combined Feature

Pre Rec
f1 Purity Pre Rec
f1 Purity Pre Rec
f1 Purity Pre Rec
f1 Purity
15 0.698 0.485 0.573 0.625 0.342 0.221 0.268 0.412 0.656 0.367 0.470 0.563 0.625 0.450 0.523 0.600
15 0.619 0.429 0.506 0.563 0.405 0.333 0.366 0.412 0.714 0.429 0.536 0.600 0.524 0.381 0.441 0.533
20 0.508 0.628 0.561 0.500 0.413 0.344 0.375 0.400 0.602 0.581 0.591 0.550 0.342 0.344 0.343 0.400
33 0.125 1.000 0.222 0.125 0.437 0.481 0.458 0.424 0.449 0.449 0.449 0.424 0.125 1.000 0.222 0.125
25 0.265 1.000 0.419 0.263 0.252 0.275 0.263 0.320 0.351 0.363 0.357 0.360 0.272 1.000 0.428 0.275
15 0.317 0.714 0.439 0.313 0.524 0.571 0.547 0.533 0.595 0.524 0.557 0.533 0.429 0.500 0.462 0.400
17 0.335 0.785 0.470 0.326 0.481 0.393 0.433 0.474 0.648 0.485 0.555 0.529 0.500 0.396 0.442 0.474
13 0.429 0.414 0.421 0.467 0.619 0.414 0.496 0.571 0.667 0.414 0.511 0.615 0.845 0.636 0.726 0.769
17 0.352 0.588 0.440 0.346 0.400 0.431 0.415 0.421 0.362 0.525 0.429 0.368 0.310 0.613 0.412 0.250
14 0.400 0.370 0.384 0.467 0.587 0.530 0.557 0.533 0.519 0.520 0.520 0.533 0.400 0.420 0.410 0.467
12 0.517 0.563 0.539 0.500 0.625 0.625 0.625 0.583 0.438 0.438 0.438 0.417 0.683 0.625 0.653 0.583
18 0.464 0.440 0.452 0.389 0.433 0.414 0.424 0.389 0.643 0.486 0.553 0.500 0.431 0.598 0.501 0.414

Table 1. Precision, Recall and f1 , Cluster Purity value according to Case-law and thenumber of sentences

finding similarities between sentences, and the results show that the performanceof N-gram depends upon the number of sentences; if the number of the sentencesin the case-law file is high, then N-gram performance is poor.
Sentence Closeness is another important feature that helps to understandthe sequential context of the sentence. The sentence following an argumentative
sentence often has a huge impact on the argument, as the meaning/context
of a sentence usually flows sequentially. The results in this table show that theperformance of Sentence Closeness is satisfactory, but still lacking in comparison
to Word2vec. The Combined feature also has an impact, as it is a combinationof Word2vec, N-gram and Sentence Closeness. The Combined feature offered
the highest f1 value for those case-law files for which Word2vec did not offer
significant results, with the exception of case-law files 02, 39 and 40. Overall,
66% of the case-law files obtained the highest f1 using Word2vec and Combinedfeature.
Furthermore, in the case of N-gram and the Combined feature, we found recall is further elevated by up to 1, but precision is very low for case-law files thathave a large number of sentences. This is because the n-gram feature is inappropriate for such case-law files. If the feature is not sufficiently discriminating
enough to distinguish among sentence categories, applying the FCM provides
equal membership probability values (or very close to equal ) for every category,essentially providing no useful information. As a result, during the process of
forming hard clusters, such sentences get equally distributed over all clusters.
During the evaluation process, this results in recall scores that are very highbecause a large number of sentences will be retrieved from many categories, but
this abundance of hits in many categories also implies that the precision will be
low.
Table 1 also presents the cluster purity value of each feature obtained foreach case-law file. Word2vec was found to play the leading role in case-law files
03, 13, 16, 30, 31, 32, 40, and 42. Sentence Closeness scored highest in four caselaw files: 16, 31, 39 and 41. However, case-law 16 and 31 tied with Word2vec.

Using clustering techniques to identify arguments in legal documents

Overall the purity values are satisfactory, except in case-law file 16 and 33 with
the Combined feature and N-gram. Case-law 16 which had 33 sentences had thelowest value (0.125) from Combined and N-gram features. Similarly, case-law 30,
which has 25 sentences, obtained 0.275. On the other hand, case-law 35, which
had 13 sentences, scored 0.726 (the highest value) using the Combined feature.
From this analysis, we can conclude that having a greater number of sentences
also affects the clustering quality negatively and that Word2vec is the dominantfeature for obtaining acceptable f1 and cluster purity values. It is apparent fromthe data in Table 1 that f1 and cluster purity are well correlated.
Overall, the results obtained from the proposed framework are promising even
if they cannot be compared with other researchers’ results. [14] had a similar
objective to our own, and obtained 60% accuracy while focusing on argumentation structure detection. However, since different performance measurements
are used, our results cannot be compared to theirs. Our approach, which can beused for any corpora, is not limited to a specific domain. Our results are muchcloser to [14] than [9], who obtained an accuracy of 42% while segmenting theargumentative sentence using Conditional Random Fields (CRF), or [11], whose
precision and recall for identifying argument structure using automatically segmented propositions were 33.3% and 50.0%, respectively. Moreover, their manually segmented propositions reached precision rates of 33.3% and recall rates of
18.2%. [22] also encountered problems dealing with ‘support’ and ‘attack’ relations. The main reason for this was that their approach was unable to identify
the correct target of a relation, especially in a paragraph with multiple claims
or reasoning chains.

7

Conclusion and Future Work

We proposed a new clustering technique for grouping argumentative sentencesin legal documents. We also proposed and implemented an evaluation procedure
for the proposed system. Moreover, we also proposed an approach to identify
the total number of arguments in a case-law document. Overall, the results that
we achieved are satisfactory. The macro f1 and average cluster purity score
for system prediction using Word2vec feature in case-law files that have 4 to 8
arguments is 0.497 and 0.499 respectively.
For future work, we propose adding more features, such as ‘semantic similarity’ to try to improve these results. Moreover, as an extension of this work weare working on the identification, in each cluster/argument, of sentences acting
either as a premises or conclusions.

References
1. Moh’d Belal Al-Zoubi, Amjad Hudaib, and Bashar Al-Shboul. A fast fuzzy clustering
algorithm. In Proceedings of the 6th WSEAS Int. Conf. on Artificial Intelligence,
Knowledge Engineering and Data Bases, volume 3, pages 28–32, 2007.

Using clustering techniques to identify arguments in legal documents
2. Ricardo Baeza-Yates and Berthier Ribeiro-Neto. Modern information retrieval, volume 463. ACM press New York, 1999.
3. James C. Bezdek, Robert Ehrlich, and William Full. Fcm: The fuzzy c-meansclustering algorithm. Computers & Geosciences, 10(2):191 – 203, 1984.
4. David M. Blei, Andrew Y. Ng, and Michael I. Jordan. Latent dirichlet allocation.
Journal of machine Learning research, 3(Jan):993–1022, 2003.
5. Peter F Brown, Peter V Desouza, Robert L Mercer, Vincent J Della Pietra, andJenifer C Lai. Class-based n-gram models of natural language. Computational
linguistics, 18(4):467–479, 1992.
6. Juan Cao, Tian Xia, Jintao Li, Yongdong Zhang, and Sheng Tang. A density-based
method for adaptive lda model selection. Neurocomputing, 72(7-9):1775–1781, 2009.
7. Jack Conrad and Khalid Al-Kofahi. Scenario analytics system, December 21 2017.
US Patent App. 15/618,920.
8. Douglass R Cutting, David R Karger, Jan O Pedersen, and John W Tukey. Scatter/gather: A cluster-based approach to browsing large document collections. In
Proceedings of the 15th annual international ACM SIGIR conference on Researchand development in information retrieval, pages 318–329. ACM, 1992.
9. Theodosis Goudas, Christos Louizos, Georgios Petasis, and Vangelis Karkaletsis.
Argument extraction from news, blogs, and social media. In Hellenic Conferenceon Artificial Intelligence, pages 287–299. Springer, 2014.
10. Anil K Jain, M Narasimha Murty, and Patrick J Flynn. Data clustering: a review.
ACM computing surveys (CSUR), 31(3):264–323, 1999.
11. John Lawrence, Chris Reed, Colin Allen, Simon McAlister, and Andrew Ravenscroft. Mining arguments from 19th century philosophical texts using topic basedmodelling. In Proceedings of the First Workshop on Argumentation Mining, pages
79–87, 2014.
12. Yanjun Li, Soon M Chung, and John D Holt. Text document clustering based on
frequent word meaning sequences. Data & Knowledge Engineering, 64(1):381–404,
2008.
13. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119, 2013.
14. Raquel Mochales and Marie-Francine Moens. Argumentation mining. Artificial
Intelligence and Law, 19(1):1–22, 2011.
15. Raquel Mochales-Palau and Marie Francine Moens. Study on sentence relationsin the automatic detection of argumentation in legal cases. FRONTIERS IN ARTIFICIAL INTELLIGENCE AND APPLICATIONS, 165:89, 2007.
16. Nidhi. Number of topics for lda on poems from elliston poetry archive. Available
at http://www.rpubs.com/MNidhi/NumberoftopicsLDA (2017-03-31), 2017.
17. Joonsuk Park, Arzoo Katiyar, and Bishan Yang. Conditional random fields for
identifying appropriate types of support for propositions in online user comments.
In Proceedings of the 2nd Workshop on Argumentation Mining, pages 39–44, 2015.
18. Radim Řehůřek and Petr Sojka. Software Framework for Topic Modelling with
Large Corpora. In Proceedings of the LREC 2010 Workshop on New Challenges
for NLP Frameworks, pages 45–50, Valletta, Malta, May 2010. ELRA. http://is.
muni.cz/publication/884893/en.
19. M.E.S. Rodrigues and L Sacks. A scalable hierarchical fuzzy clustering algorithm
for text mining. In Proceedings of the 5th international conference on recent advancesin soft computing, pages 269–274, 2004.

Using clustering techniques to identify arguments in legal documents
20. Christos Sardianos, Ioannis Manousos Katakis, Georgios Petasis, and Vangelis
Karkaletsis. Argument extraction from news. In ArgMining@ HLT-NAACL, pages
56–66, 2015.
21. Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction
to information retrieval, volume 39. Cambridge University Press, 2008.
22. Christian Stab and Iryna Gurevych. Annotating argument components and relations in persuasive essays. In Proceedings of the 25th International Conference on
Computational Linguistics (COLING 2014), pages 1501–1510. Dublin City University and Association for Computational Linguistics, August 2014.
23. Marco Lippi and Paolo Torroni. Argumentation mining: State of the art andemerging trends. ACM Trans. Internet Technol., 16(2):10:1–10:25, March 2016.
24. Xuanli Lisa Xie and Gerardo Beni. A validity measure for fuzzy clustering. IEEE
Trans. Pattern Anal. Mach. Intell., 13(8):841–847, August 1991.

8

Appendix

8.1

Appropriate Cluster Identification Algorithm (ACIA)

Let A be the system’s cluster set and the B the gold standard cluster set, respectively, having a cardinality of n: A = {a1 , · · · , an } and B = {b1 , · · · , bn }.
We define the matrix F = {fij } where fij = ai bj with ai ∈ A and bj ∈ B.
Here, F = {fij } is the f-measure value calculated by taking cluster i from A andcluster j from B.
We denote by (F )ij the matrix formed from the matrix F by removing the j thcolumn and ith row
State 1 : Initialization https://v2.overleaf.com/project/5b7d5fba95426b6bb8c66ef6
F o = (fij )n×n
Ro (−1, −1) = ∅i.e. Nodes are connected with the cost value C=0 to form a tree structure.
State 2 : From k = 0 to n, iterate. At each k step, we have F (k) (i, j) and R(k) (i, j)
Find all maximum
elements of F (k) (i, j)no(k)
Let Mk = (i, j)|fij is themaximum element of F (k) (i, j)i.e. Maximum f-measure value is selected and place in tree structure;
State 3: For each element (i, j) ∈ Mk , update routeR(k+1) (i, j) = R(k) (i, j) ∪ {(i, j)}and matrix
F (k+1) (i, j) = F (k)ijUsing clustering techniques to identify arguments in legal documents

Do it for all elements (i, j) of Mk
Stop when k = n.
i.e. Procedure repeat again for other remaining values;
State 4 : {For each route, calculate total cost}X
T CR(k) (i,j) =fij(i,j)∈R(k) (i,j)

i.e. The total cost of each route is calculated.
State 5: Select one of the maximum values
T CRo (i,j) ,and its route
Ro (i, j) = {(i1 , j1 ), · · · , (in , jn )}i.e. The route with the maximum scores is selected.
After identifying the appropriate cluster (argument) with respect to the goldstandard; an f-measure is calculated between the ith cluster of the system asrecommended by the ACIA and the j th cluster of the gold-standard. After that,the average f-measure value is calculated.