Legal Information Extraction and Entailment for
Statute Law and Case Law
Juliano Rabelo1 , Mi-Young Kim1,2 , Housam Babiker1,3 , Randy Goebel1,3 , andNawshad Farruque1,31

Alberta Machine Intelligence Institute, University of Alberta, Edmonton, AB,
Canada2
Department of Science, Augustana Faculty, University of Alberta, Camrose, AB,
Canada3
Department of Computing Science, University of Alberta, Edmonton, AB, Canada{rabelo,miyoung2,khalifab,rgoebel,nawshad}@ualberta.ca

Abstract. The problem of information overload in the legal domain
is increasing every day. In the Competition on Legal Information Extraction and Entailment (COLIEE), we tackle four challenges which areintended to encourage the development of systems and methods to alleviate that pressure: case law retrieval (Task 1) and entailment (Task 2),
and statute law retrieval (Task 3) and entailment (Task 4). For Tasks
1 and 2, we propose a pairwise paragraph similarity based approach,which ranked 4th place in Task 1 and 1st place in Task 2 among allteams participating in COLIEE 2018. For the statute law textual entailment challenge, we use a textual segmentation of legal data based on
the characteristics of statute law, then confirm textual entailment basedon negation/antonym relations. We have evaluated our system using thedata from the COLIEE 2018. The statute law competition in COLIEE
focuses on the legal information processing required to answer yes/noquestions from legal bar exams, and it consists of two phases: legal adhoc information retrieval (Task 3), and textual entailment (Task 4). We
focus on Task 4, and our experimental evaluation demonstrates the effectiveness of the legal segmentation and negation/antonym detection.
Our results show that our method was ranked first in the Task 4 of the
COLIEE 2018 competition.
Keywords: Legal textual retrieval · Legal textual entailment · Document similarity · Binary classification · Imbalanced datasets

1

Introduction

Every day, large volumes of legal data are produced by law firms, law courts,
independent attorneys, legislators, and many other sources. In that scenario,
the amount of information generated becomes manually intractable, and calls
for the development of tools which automatically or semi-automatically aid legal
professionals to handle the information overflow. In COLIEE, we face four facetsof that challenge: case law retrieval, case law entailment, statute law retrieval

Rabelo et al.

and statute law entailment. Here we provide the details of our approaches tothose problems, evaluate the results achieved and comment on future work tofurther improve our models.
For the case law retrieval (Task 1), our approach relies on measuring the
pairwise similarity between paragraphs of two case laws, generating feature vectors based on those similarities, and then using a classifier to determine whether
those cases should be noticed or not. We compare that strategy with a baseline
algorithm with uses the full text of the cases on the similarity calculation to confirm the hypothesis that considering individual paragraph similarity provides amore meaningful way of comparing case laws.
A similarity based strategy is also applied for the entailment task (Task 2).
In that case, we augment the model with word embeddings and use the available
decision and summary information, in addition to the case law contents, for thesimilarity calculation. For both tasks 1 and 2, we discuss the problem of classimbalance and propose/evaluate strategies to cope with that problem.
In the statute law information extraction, our approach to Yes/No legal question answering requires two steps. First, an information retrieval system must
retrieve relevant documents for a legal question (Task 3). Second, a textual entailment system compares the semantic connections between question and relevant
documents, and determine whether an entailment relation holds (Task 4).
Here, we focus on Task 4. First, we segment the statute law based on legal
condition/conclusion/exception, and then detect negation/antonym relation for
textual entailment. The detailed method will be explained in Section 4.

2

Case Law Retrieval

This task consists in finding which cases, in the set of candidate cases, should be“noticed” in respect to a given query case. “Notice” is a legal technical term that
denotes a legal case description that should be relevant to a query case. Moreformally, given a query case q and a set of candidate cases C = {c1 , c2 , ..., cn }, thetask is to find the supporting cases S = {s1 , s2 , ..., sn | si ∈ C ∧ noticed(si , q)}
where noticed(si , q) denotes a relationship which is true when si ∈ S is a noticed
case in respect to q.
In the following subsections, we will present an overview of the competition
dataset, discuss details of our method and analyze the results it achieved.
2.1

Dataset Analysis

The dataset provided for this task has 285 query cases, each consisting of a summary file (i.e., a short, summarized version of the case) and a fact file containing
a list of the case paragraphs. For each query case, a set of 200 candidate cases isgiven, from which must be identified a subset of the noticed cases. The candidate
cases are given in raw text, i.e., they are a full text dump of the case laws descriptions, containing not only the paragraph list, but also parties, judges, dates,
key concepts, etc. Table 1 summarizes the dataset properties.

Legal Information Extraction and Entailment for Statute Law and Case Law
Table 1. Summary for the Case Law Retrieval Task dataset.
Property
Value
Number of query cases
285
Candidate cases per query
200
Total number of candidate cases
57,000Total number of true noticed cases
2,813 (4.93%)
Average number of noticed cases per query
9.87
Standard deviation of noticed cases counts
8.8143

2.2

Our Method

Before describing the details of our approach, it is important to clarify one
aspect about our task and how one could tackle it. As noted in section 2, we are
required to identify which candidate cases should be noticed in respect to eachgiven query case. But what makes a case to be noticed in respect to another case?
Noticing a case is something a lawyer does when he/she understands that caseis somehow relevant to the case being analyzed. “Somehow relevant” is to somedegree subjective: a court is obliged to consider cases in which the same issue has
been decided in a court whose decisions are binding. Similarly, a court is obliged
to consider its own precedent. But a lawyer may also find a case is relevant andshould be noticed because it is on the same legal subject, because it addresses
similar facts, because it addresses similar parties, because it requires a similar
type of analysis, or it is for some other reason usefully analogous. Our method
to identify the noticed cases is built on an important assumption: we consider acase should be noticed if it is similar to the query case. Similarity is somewhat
vague, but seems to be a generic concept that captures most variations of thevalid reasons for noticing a case. Besides, there are several well-known existing
document similarity measures [1] that we can adopt for this task.
The Baseline Approach The baseline algorithm implemented for the caselaw retrieval task is a simple document similarity calculator: if the similarity
measured between the query case and the candidate case is above a threshold,the documents are considered similar (i.e., they should be noticed). We perform
some preprocessing on the documents before measuring the similarity: removal ofpunctuation, digits and stop words, case normalization and stemming. One of theimplemented strategies (see subsection 2.2) used a pre-trained word embedding
model, for which we did not apply stemming.
In addition to these well known preprocessing techniques, we also implemented an algorithm to join hyphenated words into a single word, since theinput documents contain hyphenated words (e.g., “attor-ney,” “deci-sion”). The
algorithm only joins two tokens separated by an hyphen if the resulting token
is a valid English word, which is determined by a dictionary lookup. That is anattempt to avoid joining two words which should really be separated by a dash(e.g., “ex-president,” “vice-consul,” “commander-in-chief”). The dictionary for
the word lookup was built by parsing an English-German dictionary [15] andRabelo et al.

parsing it to keep only the English words. The outcome of that process was an
English word list of ∼ 170, 000 terms.
In the baseline approach, the similarity between the base case and a candidate
noticed case is calculated as the cosine of the vectors formed by a bag of words
representation of each document [2]. The results achieved by this simple approachfor different threshold values are shown on Table 2.

Table 2. Baseline results.
ThresholdMeasure0.20 0.25 0.30 0.35 0.40 0.45 0.50
Recall
0.8115 0.6352 0.4827 0.3668 0.2609 0.1809 0.1038
Precision 0.0897 0.1235 0.1743 0.2459 0.3387 0.4480 0.5407
F-measure 0.1616 0.2068 0.2561 0.2944 0.2947 0.2577 0.1741

Those results show the best F-measure scores were 0.2944 and 0.2947, achieved
for thresholds of 0.35 and 0.40, respectively. The results are not great but they
provide a basis for comparison with other, more elaborate methods.

Pair-wise Paragraph Comparison Calculating the similarities using thecases full text is intuitive, but in fact there are studies showing that the mainconcepts of case laws are usually encoded in a few paragraphs [3]. To leverage that idea, we implemented a similarity based approach which compares thequery case and each candidate case paragraph-wise; thus we needed to implement a parser to read the query cases and the candidate cases, and use only theparagraph lists from both for the comparison.
That comparison step would now generate not a similarity score, but a similarity matrix. We could then use an aggregation function (such as average) on
it, which would enable us to set a threshold and make a decision on whether thecases are similar, but would also sacrifice the more detailed information, making
this approach similar to the baseline approach presented above.
Another option which would leverage the paragraph-wise similarities would
be to use that as an input to a supervised learning classifier. However, thesimilarity matrices generated have a variable number of rows and columns (since
their sizes are defined by how many paragraphs the query and the candidate caseshave). So we needed to transform each similarity matrix into a regularized input
to the classifier; this was achieved by creating a histogram over the similarity
matrices. We define the bounds of the similarity bins and count how many cells in
the matrix fall under each bin, then we normalize the histogram by the numberof cells in the matrix. This process produces a fixed size input vector which canthen be fed to a supervised learning classifier.

Legal Information Extraction and Entailment for Statute Law and Case Law

To leverage other studies [3], we also measure the paragraphs similarity considering only “legal terms”1 , i.e., we use a list of law-related terms and preprocess the paragraphs retaining only relevant terms. The similarities scores
were collected and processed in the same way as the “regular” paragraph similarities, thus augmenting the feature vectors which are then fed to the learningclassifier.
One problem which emerged was the significant class imbalance in the competition dataset (see Section 2.1): there are only 4.93% noticed cases in thedataset. To cope with that problem, we applied under-sampling over the “false”
class (i.e., the candidates which are not noticed cases). We did not want to shrink
the training dataset to only as many positive samples as there are available, so
we kept the negative dataset size five times bigger than the positive dataset.
We also attempted to oversample the positive dataset, first by replicating thepositive samples and then by synthesizing samples using SMOTE [4]. None ofthose approaches proved to be effective for this task. We also tried to use a meta
cost sensitive classifier [5], but it only made sense when we used a fully balanced
dataset for training, so we dropped that idea in favor of the use of a bigger
training dataset.
For the classification itself, we tried a few different classifiers with none or
little tweaking in the default parameters: Random Forest [6], Gradient Boosting[7] and GCForest [8]. Given the gain in performance was relevant (see Table 3),
we chose to run the rest of the experiments using only the Random Forestclassifier.
Table 3. Results for the Pairwise Paragraph Comparison Approach.
Algorithm
MeasurePrecision Recall F-measure
Random Forest
0.4200 0.3194 0.3629
Gradient Boosting 0.1422 0.3541 0.2029
GC Forest
0.3684 0.2430 0.2928

Pair-wise Paragraph Comparison Augmented with Word Embeddings
One problem of the similarity based approach is there might be paragraphs
which express similar concepts using very different phrasings. Since words withsimilar concepts should have similar vector representations in word embeddings[9], its application could help overcome that problem. The framework used in our
experiments was Google’s Word2vec [9] and the similarity was calculated using
the Word Mover’s Distance [10]: basically the smallest distance all embeddings
from one document (in our case, paragraph) need to “travel” to reach the otherdocument’s embeddings.
1

the legal terms list was created by downloading a few online available law dictionaries
and parsing their contents

Rabelo et al.

In our experiments, we used the word embeddings model pre-trained on a
general purpose set of documents [12], we figured we could have better results bytraining our own model using case law data, thus improving the representation
for case law specific terms. So we downloaded and parsed Case Law Reports [11],but that was not enough to out perform the general purpose model.

Post processing By analyzing the available dataset (Table 1), we see there
are on average 10 noticed cases for each query case; so we assumed a range
for minimum and maximum number of noticed cases for each query case. That
could help, especially in decreasing false positives if there are cases for which thealgorithm identifies too many noticed cases. But for this task, the post processing
did not help as much as it did for Task 2. Our assumption is that the much higher
standard deviation seen in the Task 1 dataset impaired the results.

2.3

Results

Below we show the results achieved by all variations of the method on the training
dataset, and the variations selected for submission on the COLIEE test dataset.

On the Training Dataset To evaluate the results on the training dataset, we
set apart 29 cases from the original 285 cases (see Table 1). Those cases were notused on the development of our method (training classifiers, manual inspection,etc). The results achieved on that validation dataset are shown on Table 4.

Table 4. Results for the Case Law Retrieval Task (validation dataset).
Algorithm
Precision Recall F-Measure
Baseline Approach
0.2609 0.3387 0.2947
Pair-wise Paragraph Comparison (PPC) 0.4200 0.3194 0.3629
PPC with word embeddings
0.4198 0.3090 0.3560
PPC with SMOTE
0.3660 0.3888 0.3771
PPC with Post Processing
0.3087 0.3784 0.3400

From those experiments, we selected to submit to the official COLIEE evaluation the results of PPC, PPC with SMOTE and PPC with Post Processing.
PPC with word embeddings did not provide improvements on the F-measure, incomparison with PPC, and was computationally quite expensive.

On the COLIEE Test Dataset The results achieved by the three selected
methods are presented on Table 5.

Legal Information Extraction and Entailment for Statute Law and Case Law
Table 5. Results for the Case Law Retrieval Task (real COLIEE test dataset).
MethodPrecision Recall F-Measure
Pair-wise Paragraph Comparison (PPC) 0.3724 0.3227 0.3458
PPC with SMOTE
0.3538 0.3926 0.3722
PPC with Post Processing
0.3484 0.4038 0.3740

2.4

Future WorkWe plan to explore other alternatives for this task. The result analysis showed
the similarity based features do provide relevant information for the classifiers
but were not great discriminants for the problem. To overcome that problem, wemight extract other features. One alternative is to extract LWIC features [13].
A more immediate option is the use of the summary data (we only used the caselaw main contents in our experiments).
We also intend to perform other experiments with word embeddings. Although the results achieved in our experiments were not so promising, we believe
the idea may be refined. The first thing would be to use specific-domain data to
train the model. We do have some data available, which has been downloaded
from [11], and we could augment it with the 57,000 candidate case laws provided
as this task’s training dataset. Another slightly different option is to re-train ageneral purpose word embedding model with our data. We believe those alternatives would improve the vector representations of the model for the problem athand. We could also try coarser grained embeddings (sentence or document level
embeddings [20], as opposed to the current word-based implementation). However, we would need significantly more data to train those embeddings, whichmakes it a poor fit for the COLIEE scenario, where relatively little data is available. A more viable alternative would be the application of different frameworks,
such as GloVe [16] or Fasttext [17], which may capture better vector representations in our scenario, or trying a different learning model for the embeddings.
We used the default CBOW model, but an option worth trying would be to use
skip-gram (which usually works better than CBOW for smaller datasets [18]).
One of the major problems which impacted our results was the severe classimbalance. For the training to be performed reasonably, we needed to balance thetraining dataset with undersampling/oversampling. Still, the classifier is applied
to a real test dataset which presents the original imbalance. Overcoming thatsituation requires modeling the problem in a different way. One option would
be modeling it as outlier detection. We could train a one-class classifier (e.g.,
One Class SVM [19]) with the false samples (the more abundant class). Samples
classified as not belonging to that class would be considered true noticed cases.
Another alternative would be applying clustering to the training dataset, possibly allowing multiple clusters for each class to capture diversity in the training
data. New samples would be classified with the class of the nearest clusters.
A more sophisticated alternative would be the extraction of the ratio decidendi [21] for each case. If we can successfully implement that and an effective
similarity measure between ratio decidendis, it would be simple to establish someRabelo et al.

sort of threshold to determine which cases should be noticed. However, the single previous description on automatically extracting the ratio decidendi [21] hasbeen tested on only one case law; the technique may not be adequate to an open
world scenario. Extracting a ratio decidendi means fully understanding a caselaw, and implementing a general approach to that problem is not somethingalready solved in the literature. But in general, the development of methods forfully understanding case law is precisely the ultimate goal of Task 1 on COLIEE.

3

Case Law Entailment

Given a base case and its respective decision, and a second case which is relevantin respect to the base case, this task consists in determining which paragraphsof the second case entail the decision of the base case. More formally, given abase case b and its decision d, and another case r represented by its paragraphs
P = {p1 , p2 , ..., pn } such that noticed(b, r) as defined in section 2 is true, weneed to find the set E = {p1 , p2, ..., pm | pi ∈ P } where entails(pi , d) denotes a
relationship which is true when pi ∈ P entails the decision d.
In the following subsections, we will present an overview of the provided
dataset, discuss details of our method and analyze the results it achieved.
3.1

Dataset Analysis

The dataset provided for this task has 181 base cases, each consisting of a summary file, a fact file (a list of the case paragraphs), and a decision file. For eachbase case, it is provided a respective related case represented by a list of paragraphs, from which the paragraphs that entail the base case decision must be
identified. Table 6 summarizes the dataset properties.
Table 6. Summary for the Case Law Entailment Task dataset.
Property
Value
Number of base cases
181
Total number of paragraphs in the related cases
8,794
Total number of true entailment paragraphs
239 (2.71%)
Average number of entailment paragraphs per base case
1.32
Standard deviation of entailment paragraphs counts
0.7106

3.2

Our Method

This task is even more challenging than Task 1. Solving it would ideally requirein-depth domain and common sense knowledge, plus a reasoning capability inorder to mimic the whole process undertaken by a human annotator who analyses the data to determine which paragraphs from one case entail a decision ofLegal Information Extraction and Entailment for Statute Law and Case Law

a different case. Assuming we do not currently have tools to implement such acomplex model, we decided to try simpler approaches which do not attempt toreason and come to a conclusion. Our method tries to extract relevant features
which (hopefully) correlate with the entailment relationship, and then train aclassifier which is capable to learn that relationship. We also face the same problems of data scarcity and class imbalance we faced in Task 1, but those problems
are more extreme for Task 2. Table 6 shows we have less data for this task andclass imbalance is more severe. We experimented three oversampling techniques
to cope with the imbalance problem: besides simple replication and SMOTE (seesection 2.2), we also implemented a new instance synthesis algorithm: for every
instance p in the positive dataset, compute a new instance of p as µp + p.
Preprocessing The preprocessing used here was a little more elaborate that
it was for Task 1: in addition to using the simple techniques used for Task 1(case normalization, stop words, punctuation, accents and digit removal, andhyphenated words joining), here we also implemented a language detection algorithm. This was necessary because the dataset contains many paragraphs written
in French2 . Since our approach is based on the similarity between documents,
and the input cases are in English only, we decided to not consider the French
candidates, although there are 6 expected answers which are identified by ouralgorithm as French written paragraphs for Task 2. This was considered an acceptable limitation of our method, since those cases represent only 2.5% of thetotal positive samples. Moreover, the task is already hard enough, so we decided
not to tackle cross language entailment for now. After the removal of the French
paragraphs, we could use the English instances to train our model3 .
Looking for Features Our first attempt relied solely on searching for words
which could indicate a paragraph was written in a way the judge would be expressing his/her decision. This would not take into consideration any relationshipwith the base case, but instead would only look for “decision-like” paragraphs.
It was not expected that this would be the final method for Task 2, but onlya means of verifying the correlation between some words and the task goal. In
fact, only by looking up a few words such as first person pronouns (“I,” “my”
etc.), we could identify around 25% of the entailment paragraphs (a good recall
for such a simple approach). However, the precision was around 2%, meaning
this method alone was far from ideal (our f-measure would be 0.0370). However,it showed there might be words correlated with the entailment paragraphs.
Word Clouds A slightly more sophisticated approach in that direction was
based on the generation of word clouds for the true and false samples. We divided
2
3

our algorithm could identify 145 instances of French paragraphs in the dataset
the official COLIEE test dataset did not present any cases of French paragraphs.
Still, removing those paragraphs was important because we used the training dataset,
which does contain French paragraphs, for training our classifiers.

Rabelo et al.

all the candidate paragraphs according to their label (true entailment or not)and generated an aggregate word cloud (or bag of words) for each class. Then,
we took the first n words from each cloud, removed duplicates, and generated
a dictionary. Each paragraph in the related cases were then encoded as one-hot
vectors according to that dictionary; the resulting vectors were fed to a classifier.
Since the resulting vectors were very sparse, we augmented this approachwith word embeddings: for a word w in the paragraph which was not in the
dictionary, we looked up the most similar word on the embeddings (i.e., theclosest concept) and used the similarity score as the feature for w in our feature
vector. Therefore the feature vectors were not one-hot encoded, being now in
the [0 − 1] range. Results on the evaluation dataset are detailed on section 3.3.
The Similarity Based Approach Since those approaches did not provide
acceptable results, our next try was based on the similarity method presented
on section 2. We measured the similarity between each paragraph in the noticed
case and the base case, generating a feature vector comprised of the measures ofsimilarity between the paragraph and (1) the decision of the base case, (2) thebase case summary and (3) the base case paragraphs (actually a histogram ofthe similarities between each noticed case paragraph and all paragraphs from thebase case). In the end, we had one feature vector representing each “candidate”
paragraph (i.e., a paragraph in the noticed case, which in this task is a candidate
for entailing the decision on the base case). That approach provided much better
results. We ran experiments using Random Forest [6] and Gradient Boosting [7]classifiers. See section 3.3 below for details.
LIWC Features In an attempt to further improve those results, we augmented
the feature vectors with LIWC features [13]. Those are features usually appliedin medical (psychiatric/psychological) applications [14] and include long words
count, pronouns count (aligned with the idea in our first approach), sentiment
analysis related features and many others. Those features did provide relevant
improvements on the model performance, as shown on subsection 3.3.
Post Processing Our final attempt to improve the results consisted in postprocessing the results. Since the model receives only a feature vector representing
a candidate paragraph, it has no built-in schema for handling how the cases are
structured and how many responses for each case would be reasonable. So it could
classify 30 paragraphs as positive entailment instances for one case and identify
none for another case. Analyzing the priors in the provided dataset (see 6), we
can see that, on average, there are ∼ 2 entailment paragraphs for each case. Thus
we tried to post process the results and keep at most 5 entailment paragraphs
for cases which had many candidates identified as true samples, and at least 1
paragraph for cases which had no candidates identified as true samples. To selectwhich paragraphs to keep, we used the classifier confidence score. That provided
a relevant increase in the model performance, especially for the precision score,
as it is shown on table 7 (subsection 3.3).

Legal Information Extraction and Entailment for Statute Law and Case Law

3.3

Results

We detail below the results achieved by all variations of the method on the
training dataset, and the three variations submitted on the COLIEE test dataset.
On the Training Dataset To evaluate the results on the training dataset,
we set apart 25 cases from the original 181 cases (see Table 6). Those cases
where not used on the development of our method (training classifiers, manual
inspection, etc). Table 7 shows the results on that validation dataset.
Table 7. Results for the Case Law Entailment Task (validation dataset).
Algorithm
Precision Recall F-Measure
Word cloud
0.080 0.133
0.100
Similarity based
0.108 0.333
0.163
Similarity based + Word cloud
0.074 0.467
0.128
Similarity based + LIWC
0.238 0.167
0.196
Similarity based + Post processing
0.2424 0.2857 0.2622
Sim. based with Grad. Boosting + Data augmentation
0.7
0.25
0.3684

As a comparison baseline, we might consider a naive classifier which outputs
all paragraphs as entailing the corresponding decisions, which would have the following scores in our validation dataset: 0.0291 (precision), 1.0 (recall) and 0.0428(f-measure). The other naive classifier option (which would output false for all
candidate paragraphs) would have a recall score of 0.0 and, as a consequence,an f-measure of 0.0, and thus is not a viable benchmark.
From Table 7, the best results our model attained were achieved using thesimilarity approach with a Gradient Boosting classifier and data augmentation(see subsection 3.2). However, we finished implementing it after the COLIEE
dry run deadline, so we did not submit a formal result based on that approach.
Not considering it, the best results were attained by combining the similarity
based approach (with a Random Forest classifier) and post processing. That improved precision considerably and, despite a drop in recall, the F-measure had asteady improvement of 60.85% (from 0.163 to 0.2622). The word cloud approach
showed poor results when used by itself, and even impaired the results of thesimilarity based approach when combined with it. Augmenting the similarity
based approach with LIWC features provided a relevant increase of 20.24% on
the F-measure. We did not have time to test post processing on that approach.
On the COLIEE Test Dataset Since we achieved the best results with thesimilarity based + post processing approach (see Table 7), we submitted three
results based on it (acronym SB+PP), varying solely the number of estimators
for the Random Forest. The results on the test dataset are shown on Table 8.
The best F-measure achieved was 0.258, which is still quite low and evensomewhat disappointing. Nonetheless, that was the best performance among

Rabelo et al.
Table 8. Results for the Case Law Entailment Task (real COLIEE test dataset).
Algorithm Number of estimators Precision Recall F-Measure
SB+PP
250 (default)
0.238095 0.283019 0.258621
SB+PP
100
0.190476 0.226415 0.206897
SB+PP
500
0.238095 0.283019 0.258621

all teams which submitted results for Task 2 in COLIEE 2018. The F-measurefor the similarity based approach with a Gradient Boosting classifier and data
augmentation was 0.12698, much lower than what we had on the validation
dataset. That probably was due to a model over-fitting.
3.4

Future WorkThe results presented above indicate this task has a high degree of difficulty,
and shows the methods we used to tackle the problem must be refined. In fact,
it may be more promising to devise completely new approaches. Applying wordembeddings is still worthy for this task, but we need to improve the vector
representations. We plan to use here the options cited on section 2.4: train aword embedding model with our own data, retrain the used model, test skipgram and experimenting with other frameworks. It is worth trying to refine theuse of the Gradient Boosting classifier, which achieved the best results on ourvalidation dataset. With more time, we can investigate and configure the model
to better generalize. However, we think the real gain in performance for Task 2
will come from a better understanding of the problem. We need to study what
exactly makes an entailment relationship hold between case laws and then devise
more effective strategies to model that relationship.

4

Statute Law Entailment

4.1

MethodFor the statute law entailment, we used the characteristics that a legal statuteconsists of “general condition,” “conclusion,” “exceptional case,” and “conclusion for the exceptional case.” The query can belong to either “general condition” or “exceptional case.” We filter out the cases in the statute that are notrelated to the input query, by figuring out where the input query belongs. We
use condition/conclusion/exception detection rules defined in Kim et al. [22].
We determine if a query belongs to the general case or exceptional caseby computing the proportion of shared words between query and condition,and query and exception condition. We collect words from the detected condition (exception condition) and conclusion (exception conclusion), and also detect negation value for each condition (exception condition) and conclusion (exception conclusion): The negation level (neg level(segment)) is computed as following: if [negation + antonym] occurs an odd number of times in the segment, its negation level is 1. If [negation + antonym] occurs an even (or zero)Legal Information Extraction and Entailment for Statute Law and Case Law

number of times, its negation level is 0. We consider the entailment holds if
neg level(condition of the relevant article) + neg level(conclusion of the article)
is the same to neg level(condition of the query) + neg level(conclusion of thequery). Otherwise, we consider the entailment does not hold. To detect negation
and antonym, we manually constructed negation/antonym dictionary from thetraining data.
4.2

Experimental Results

The dry run data size of COLIEE 2018 is 644 queries for Task 4 from theJapanese bar exam from 2006 to 2016, and the formal run data size is 69 queries
from the bar exam of 2017. Our performance of textual entailment is 63.77%,
which was ranked No.1 in the COLIEE 2018 competition as shown in Table 9.
Table 9. Experimental results on formal run data of COLIEE 2018
Participating Team Accuracy(%)
UA(our team)
0.6377KIS Frame
0.5652
KIS mo30.5507
KIS dict
0.5362KIS SVM
0.5217KIS Frame2
0.5072
UE
0.4783

5

Conclusion

The similarity based approach used for tasks 1 and 2 provided reasonable resultson the COLIEE testing datasets. In Task 1, we ranked 4th place, whereas in
Task 2 we ranked 1st place among all teams in COLIEE 2018. However, theabsolute scores achieved show our method still needs to be improved before it
could be used in real world applications. For the statute law competition, we
proposed a method to answer yes/no questions from legal bar exams related tocivil law statutes. We performed legal segmentation of the statute law, and usednegation/antonym information for textual entailment. The performance of oursystem was ranked 1st place in Task 4 of the COLIEE 2018 competition.

Acknowledgments
This research was supported by Alberta Machine Intelligence Institute (AMII),
and would not be possible without the further support of Colin Lachance fromvLex and Compass, and the teams from Ross Intelligence and Intellicon.

Rabelo et al.

References
1. Lee, M. D., Pincombe, B., Welsh, M.: An Empirical Evaluation of Models of Text
Document Similarity. In Proceedings of the Annual Meeting of the Cognitive Science
Society, vol. 27, pp 1254–1259 (2005).
2. Baeza-Yates, R. A., Ribeiro-Neto, B.: Modern Information Retrieval. AddisonWesley Longman Publishing Co., Boston, MA, USA (1999).
3. Kumar, S.: Similarity Analysis of Legal Judgments and applying Paragraph-link to
Find Similar Legal Judgments. Master thesis. Hyderabad, India (2014).
4. Chawla, N., Bowyer, K., Hall, L., Kegelmeyer, W.: SMOTE: Synthetic Minority
Over-sampling Technique. Journal of AI Research, v. 16, 321–357 (2002).
5. Sun, Y., Kamel, M. S., Wong, A., Wang, Y.: Cost-sensitive boosting for classificationof imbalanced data. Pattern Recognition, vol. 40 (12), 3358–3378 (2007).
6. Breiman, L.: Random Forests, Machine Learning, 45(1), 5–32 (2001).
7. Friedman, J. H.: Greedy Function Approximation: A Gradient Boosting Machine.
Annals of Statistics, vol. 29, pp. 1189–1232 (2000).
8. Zhou, Z., Feng, J.: Deep Forest: Towards An Alternative to Deep Neural Networks.
In Proceedings of the Twenty-Sixth International Joint Conference on Artificial
Intelligence, pp. 3553–3559 (2017).
9. Mikolov, T., Sutskever, I., Chen, K., Corrado, G., Dean, J.: Distributed Representations of Words and Phrases and Their Compositionality. In Proceedings of the
26th International Conference on Neural Information Processing Systems, vol. 2,
pp 3111–3119, Lake Tahoe, NV, USA (2013).
10. Kusner, M., Sun, Y., Kolkin, N.I., Weinberger, K.: From word embeddings to document distances. In Proceedings of the 32nd International Conference on Machine
Learning (ICML), pp 957–966 (2015).
11. Supreme Court of Canada, scc-csc.lexum.com. Accessed on Sep 4th, 2018.
12. Google
Pre-trained
Word
Embeddings
Model,
https://code.google.com/archive/p/word2vec/. Last accessed on Sep 4th, 2018.
13. Pennebaker, J. W., Booth, R. J., Francis, M. E.: Linguistic Inquiry and Word
Count: LIWC [Computer software]. Austin, TX: LIWC.net. (2007)
14. Tausczik, Y. R., Pennebaker, J. W.: The Psychological Meaning of Words: LIWC
and Computerized Text Analysis Methods. Journal of Language and Social Psychology 29(1), pp. 24–54 (2010).
15. Online-Wörterbuch Englisch-Deutsch, www.dict.cc. Accessed on Sep 4th, 2018.
16. Pennington, J., Socher, R., Manning, C.: GloVe: Global Vectors for Word Representation. Empirical Methods in Natural Lang. Processing. pp. 1532–1543 (2014).
17. Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching word vectors withsubword information. Transactions of the Association for Computational Linguistics, vol. 5, pp. 135146 (2017).
18. Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In Proc. Int. Conf. on Learning Representations (2013).
19. Manevitz, L. M., Yousef, M.: One-class Svms for Document Classification. The
Journal of Machine Learning Research, vol. 2, pp. 139–154 (2002).
20. Le, Q., Mikolov, T.: Distributed Representations of Sentences and Documents. In
Proc. of the 31st International Conf. on Machine Learning, Beijing, China (2014).
21. Valvoda, J., Ray, O.: From Case Law to Ratio Decidendi. New Frontiers in Artificial
Intelligence. pp. 20–34. Springer International Publishing (2017).
22. Kim, M.-Y., Xu, Y., Lu, Y., and Goebel, R.: Question Answering of Bar Exams
by Paraphrasing and Legal Text Analysis. In Lecture Notes in Artificial Intelligence(JSAI International Symp. on Artificial Intelligence), Vol.10247:299-313, (2017).