Case law retrieval with doc2vec
and Elasticsearch⋆
Wilco Draijer and Suzan Verberne
Leiden Institute of Advanced Computer Science,
University of Leiden, Niels Bohrweg 1, Leiden, The Netherlands
w.j.w.draijer@umail.leidenuniv.nls.verberne@liacs.leidenuniv.nl

Abstract. In this paper a Random Forest Classifier with eight different
features is implemented and evaluated as legal case retrieval system for
COLIEE 2018. Based on a summary of and facts about a new casethe system makes a prediction using TF–IDF, doc2vec and the More
Like This score from Elasticsearch about the relevance of candidates.
The system scored third out of twelve submissions in the competition.
The precision is competitive, but the recall is fairly low. We found that
TF–IDF and the More Like This score are the most important predictorsof relevancy. Lowering the minimum document frequency of TF–IDF
does not improve the results. As future work we propose noun phrase
chunking as an extension of this system to retrieve the full name ofmultiple acts (e.g. ‘the immigration and refugee protection act’).
Keywords: Legal Information Retrieval · doc2vec · Elasticsearch

1Introduction

The Competition on Legal Information Extraction/Entailment (COLIEE) [1]is an annual international competition on legal information retrieval. COLIEE
2018 exists of four separate tasks.
The first task involves case law retrieval. A number of supporting cases (onaverage ten) from 200 candidates are to be extracted for each of 285 new cases.
The new cases have a short summary and corresponding facts. The candidates
are not the same for every new case.
The second task is about entailment. A decision in a new case is based on arelevant old case. The aim of this task is to find the paragraph of the old caseon which the decision is based.
The third task is finding relevant Japanese Civil Code Articles for a legal bar
exam question. The answer is a number of articles for every question.
The fourth and final task is to identify relevant articles for questions.
This paper describes our participation in only the first task of thecompetition. The built system is a random forest classifier based on eight⋆

Supported by ORTEC – Optimize Your Worlddifferent features: four features based on TF–IDF, two features based on
doc2vec, and two features based on a more-like-this query in Elasticsearch. We
obtained satisfying results; our system ranked third out of twelve submissions
from six different teams with an F1-score of 0.393.
The remainder of this paper is organized as follows: Section 2 gives
background on the data the system needs to handle and provides a problem
description. Section 3 describes the methods used to build the system. Section
4 describes the experiments we have done. Section 5 provides the results of thesystem, which are then discussed in section 6. Section 7 concludes the paper
and makes suggestions for future work.

2

Data and Problem Description

For the case law retrieval task 285 new cases were given as training set. Eachcase has a textual summary describing the case, and a list of facts about thecase. An example of a summary and facts of a new case is given in appendix A
and B. The test set existed out of 59 new cases, also with 200 candidates.
Each of the new cases had 200 prior cases as candidates. The task at hand
was to select for each new case which of these 200 candidates are relevant for the
new case. There were between 3 and 16 relevant candidates for each case. The
distribution of word counts of the candidate cases, the facts and the summary are
somewhat skewed to the right side. This is shown in table 1, where the average,
median and maximum word count, and standard deviation are given.
Table 1. Documents word countType Median Average Std. Deviation Maximum word countCandidates 24,654
35,53337,390.98
516,321
Facts
5,675
6,772
4,549.85
33,113
Summary426
538
439.12
5,247

This shows that the candidates are significantly longer and presumably holdmore information than the provided facts and summaries for the new cases.

3

Used Methods

Since classifiers are not able to handle text data directly, we have to engineer
numeric features based on the text of the cases. If these numeric features area good representation of the content of the cases, decision trees will be able tosegregate the cases based on these numbers grouping similar cases together. A
random forest classifier trains a number of decision trees, which will all vote on
the classification of a case. The numeric features we want to extract from thecandidate cases are based on three methods: TF–IDF, doc2vec and More Like
This. In this section we will introduce all three methods.

3.1

TF–IDFTo find distinct characteristics of a document just counting words in a single
document will not necessarily give a fair insight. Even after removing stop–words
specific words appear very often in a document. If such words appear often inall documents it will not be a distinctive term. For example, in this task theword ‘case’ was frequently found in all candidates and summaries. This does notdistinct any particular candidate.
To account for this we used Term Frequency – Inverse Document Frequency(TF–IDF) [2]. This metric indicates how often a word occurs in a document,
weighted by the number of documents the word occurs in.
We use the implementation of the TfidfVectorizer in the sklearn package in
Python. We create a TF–IDF vector for all documents to find similarity between
documents. If the euclidean and/or cosine distance between vectors is small it
shows that the documents have similar specific word counts which indicates thatthey have similar contents.
3.2

Doc2vec

Le and Mikolov [3] proposed an unsupervised framework inferring continuous
distributed paragraph vector representations for documents, called doc2vec,
which is an extension of the word2vec model [4]. Word2vec learns linguistic
context of words by training neural networks with a large corpus of text. Doc2vec
extends this by adding a paragraph ID to the vectors. Where word2vec learns
a numeric representation of words, doc2vec learns a numeric representation of
documents just by adding this paragraph ID. Doc2vec gained popularity overthe last few years as it is useful for embedding larger blocks of text.
Similar documents will have similar doc2vec vector representations. Therefore
we added a doc2vec vector to all documents and calculated the distance between
the vectors to identify similarity.
3.3

More Like This

Elasticsearch [5] is an efficient search and analytics system. One of thefunctionalities in their search is the More Like This-query (MLT). It takes atext field, searches for similar texts and gives a score to all results. It uses the
Lucene Scoring Formula [6], which uses TF–IDF. The Lucene Scoring formulais calculated as follows:score(q, d) = coord-f actor(q, d) · query-boost(q)V (q) · V (d)· doc-len-norm(d) · doc-boost(d)·|V (q)|With:– q as the query(1)

– d as a document– coord-factor(q,d): When multiple items are queried, this factor may reward
documents extra for every additional term matched– query-boost(q): the user can specify boosts to each query, sub-query, andeach query term which will receive extra weight– V(q) · V(d): the dot product of the weighted vectors (TF–IDF)– |V(q)|: Euclidean norm
– doc-len-norm(d): document length normalization factor– doc-boost(d): the user can specify important documents which will receive
extra weight
The biggest difference between More Like This and the regular TF–IDF is
that More Like This always normalizes on document length. Though relevanceclassification tends to favor longer documents, since they contain more words
and therefore could be unfairly advantaged), documents holding absolute more
relevant information could be more interesting than those which hold relatively
more relevant information.

4

Experiments

The system that is built uses Elasticsearch as a database. Every case has thefollowing different fields: original text, preprocessed text, TF–IDF distance score,
doc2vec vector representation.

4.1

Preprocessing

To make the original text more usable multiple preprocessing steps are
implemented.
1. Lowercased entire text
2. Removed additional info like the numbered tags for the paragraphs
3. Removed all signs and punctuation marks
4. Replaced ‘s. 4’, ‘sec. 4’ or ‘section 4’ with ‘section 4’ (for all numbers andwith or without dot after s or sec)
5. Step 4 but with ‘art.’ or ‘article’ and ‘topic’
6. Replaced ‘subject act’ with ‘subject act’
7. Removed all stop words (using the nltk stop words package)
8. Removed all words with two or less characters
9. Lemmatized all words (using the nltk WordNetLemmatizer)
This preprocessing made the texts less complex, without removing essential
words or phrases.

Table 2. Features
FeaturesMore Like This Score on Facts
More Like This Score on Summary
Doc2vec Cosine Similarity distance to Facts
Doc2vec Cosine Similarity distance to Summary
TF–IDF Euclidean distance to Facts
TF–IDF Euclidean distance to Summary
TF–IDF Cosine similarity distance to Facts
TF–IDF Cosine similarity distance to Summary

4.2

Feature Extraction

We extracted eight features from all candidate cases. These features are shownin table 2.
TF–IDF is implemented for unigrams and a minimum document frequencyof 400. We decided to create a new TF–IDF corpus for the test set and not use
the one from the training set. The dimensionality of the TF–IDF vectorizer is
9,147 for the training set and 3,444 for the test set. Doc2vec is implemented witha vector size of 100. These features can be used as an input for a classifier.

4.3

Distance Metrics

TF–IDF and doc2vec are both vectors. The distance between two vectors can bemeasured in multiple ways. The most straightforward way is euclidean. This isa straight line distance measurement. Cosine similarity takes the angle between
two vectors. These distances are shown in figure 1, where d is the euclidean
distance and θ the cosine similarity.
When two documents have similar directions (topics), but not similar lengths(occurrences of certain words), the euclidean distance measure will be large while
the cosine similarity not. If direction is a better indicator than length cosine
similarity should be used. In this system the distance for doc2vec is measured
with cosine similarity, TF–IDF is measured with cosine similarity and euclidean
distance (as two different features).

4.4

Random Forest Classifier

The system uses a Random Forest Classifier, which uses the features describedin table 2. It is trained and tuned with five fold cross validation to find the bestparameters for the system. The chosen parameters based on this training areshown in table 3.

Fig. 1. Euclidean Distance (d) and Cosine Similarity (θ)

Table 3. Random Forest Classifier Parameters.
Parameter

Value

Researched Range

Number of trees
Criterion
Minimum samples to split anode
Maximum depth
Minimum samples in a leaf
Maximum features in a tree

45
Gini (impurity)2

15–85 in steps of 5
Gini and Entropy
2–12 in steps of 1

35 nodes2
Square root of number offeatures5
Precision, recall, f-measure
F-measure

10–80 in steps of 5
2–10 in steps of 1
1–8 in steps of 1 andsquare root
3 and 5

Cross-validation folds
Scoring methods
Refit scoring methods

5

Results

The test data is classified by the system and COLIEE has sent the results of theclassification back; these are shown in table 4, sorted by F-measure. Our teamuses the id ‘UL’.
Table 4. Results
idPrecision
RecallF-measure
JNLP-k=100.676271 0.634340
0.654635
JNLP-r=2.5 0.546419 0.655008
0.595806
UL
0.563798 0.302067 0.393375
HUKB1
0.497436 0.308426
0.380765
UA-postproc 0.348422 0.403816
0.374080
UA-smote
0.353868 0.392687
0.372268
HUKB2
0.404661 0.303657
0.346957
UA
0.372477 0.322734
0.345826
Smartlaw0.287076 0.430843
0.344565UBIRLED-2 0.195511 0.720191
0.307536UBIRLED-1 0.132881 0.623211
0.219056
UBIRLED-3 0.561404 0.101749
0.172275

5.1

Feature importance

The Random Forest Classifier attributed a feature importance to all features.
These are shown in table 5.
Table 5. Feature ImportanceFeatureMore Like This Score on Facts
TF–IDF Euclidean distance to Facts
TF–IDF Cosine similarity distance to Summary
TF–IDF Cosine similarity distance to Facts
TF–IDF Euclidean distance to Summary
More Like This Score on Summary
Doc2vec Cosine similarity distance to Summary
Doc2vec Cosine similarity distance to Facts

Importance
0.18680.14590.14520.13350.11160.11030.08380.0829This shows that a doc2vec vector representation of a document is lessdistinguishing than the other features.

5.2

Improvements

Based on these results we tried to improve the system by choosing a better
minimum document frequency for TF–IDF, by grouping the train and test set
in the corpus for TF–IDF, by shortening the length of the doc2vec vectors andby removing less important features. The tuning of the hyperparameters of thesystem using new or other features were researched the same way as shown intable 3 The results of the new improvements are shown in table 6.
We tried to improve the TF–IDF by decreasing the Minimum Document
Frequency to 250, 40 and 4 and by grouping the corpora from the train and test
set; improve doc2vec by decreasing the length to 50 from 100; and improve thesystem overall by removing doc2vec, since it is less distinguishing.
Table 6. Potential Improvements
Improvement
Base line
Minimum Doc. Freq.
TF–IDF
Doc2vec length
Removing lessimportant features

Change/Value440250Grouped corpora
50
No Doc2vec
No eucl. and cosine
to summary

Precision
0.56380.57360.56200.54150.59700.52530.55370.4554

Recall0.30210.23530.24480.27980.2496
0.29730.3196
0.2925F-measure
0.39340.33370.34110.36900.35200.37970.40520.3562This shows that the F-measure does not improve with a reduced minimum
document frequency; especially the recall reduces. Grouping the train and test
set in the same TF–IDF corpus increases the dimensionality of TF–IDF to 10,365
and improved the precision a bit, but reduced the recall. Altering the doc2vec
length to 50 has a slight decrease in F-measure.
When the doc2vec features are removed, the F-measure increases a bit. The
new feature importance changes are shown in table 7. Since the euclidean andcosine distance to the summary are the new least important features, these were
to be removed next. This results in a substantial decrease in F-measure.
5.3

Distance Importance

When the doc2vec features are removed, the system is in the end only based on
TF–IDF with different distance measures. Since this seems to be redundant weresearch the impact of only using a subset of these features. Table 8 shows theseresults. This shows that though More Like This has a higher feature importance(especially the MLT score on Facts), it does not perform well when used without
the other distance measures.

Table 7. New Feature ImportanceFeatureMore Like This Score on Facts
TF–IDF Cosine similarity distance to Facts
TF–IDF Euclidean distance to Facts
More Like This Score on Summary
TF–IDF Cosine similarity distance to Summary
TF–IDF Euclidean distance to Summary

Importance
0.23640.17980.16920.14330.13720.1342

Table 8. Distance Comparison
Distances used
Cosine, Euclidean & Normalized Euc (MLT)
Cosine & Euclidean
Euclidean & Normalized Euc (MLT)
Cosine & Normalized Euc (MLT)
Only Cosine
Only Euclidean
Only Normalized Euc (MLT)

6

Precision
0.55370.3741
0.48860.51050.43610.38390.4000Recall
0.31960.32590.30680.31000.33070.31800.2003F-measure0.4052
0.34830.37700.38580.37610.34780.2669Discussion

Compared to the other teams the results from table 4 are reasonably good.
The precision is good, but the recall is relatively low. In the legal field recall
is especially important, since retrieving relevant cases is more important thancorrectly classifying as irrelevant cases. To formulate it differently: a systemthat classifies all cases as irrelevant has a very good precision, but is completely
useless.
The difference in feature importance is quite interesting. The two doc2vec
features are less important, but the other features are not clearly separable bycharacteristic: cosine similarity is not more important than euclidean, or More
Like This than TF–IDF. The Summary features do not all perform better orworse than the Fact features. There is also no distinction in using Euclidean
or Cosine similarity, which results in no clear preference for using one over theother. Using a subset of distance metrics showed it was best to use all. The factthat the feature importances all are fairly equal implies that all TF–IDF andMore Like This features are used as predictive characteristics.
6.1

Potential Improvements

The attempt to refine the F-measure by changing multiple settings resulted in
one actual improvement. Removing the doc2vec features increased the F-measurea bit. Removing even more relatively unimportant features does not result in
more improvement. The right length for a doc2vec vector differs per problem.

The results show that a length of 100 is closer to the right length than 50.
Grouping the train and test set had a small negative impact on the result. Thismeans that in this case training the Random Forest Classifier with one set andusing is with another does not negatively impacts the system. This is presumably
because only the distance between vectors matter, not about the actual vectors.
Using a subset of distance measures gives unexpected results. When using
one metric More Like This scores the worst out of the three, but when using two
the one without More Like This scores worse than the other two. In text mining
and information retrieval the cosine distance is most popular. When using only
one metric it is the best choice, as shown in table 8, but using all three gives abetter result.

7

Conclusion

We proposed the system described in this paper for COLIEE 2018.
Overall the Random Forest Classifier works fine, although it leaves room
for improvement. The precision of the system is good, but the recall is fairly
low. The feature importance, table 5, shows that the More Like This score andthe two distance metrics for TF–IDF are all similarly important for predicting
relevancy of candidates. The doc2vec features were not a good indicator, thesystem performed better when they were removed.
The reason for the reasonable F-measure our system reached can be found in
two distinctive things: the distance metrics and preprocessing. All our features
are in the end based on TF–IDF (when removing doc2vec), but used different
distance metrics (Euclidean, Cosine and More Like This used normalized
Euclidean). We showed the system performs best when using all three, butwhen using one cosine is best to be used (as is the baseline in the field of text
mining).
There are multiple options for improvement of the system. First, morefeatures could be added. A topic model like LDA could be added, following
the reasoning that similar documents will have similar topics. Second, multiple
cases contain an act existing out of multiple words, e.g. ‘the immigration andrefugee protection act’. The current preprocessing changes this to ‘immigration
refugee protection act’. To keep the act as a whole as one term noun phrase
chunking should be applied. The downside of just applying this is losing some
information, because if ‘the immigration and refugee protection act’ is changed
to ‘the immigration and refugee protection act’ it loses the individual words.
This could be prevented by adding both the individual words as the whole nounphrase, which will increase the dimensionality of the vectors. More research isneeded to investigate the value of these solutions.

References
1. COLIEE, https://sites.ualberta.ca/~miyoung2/COLIEE2018/. Last accessed 3
Sept 2018
2. Salton, G., Buckley, C.: Term-weighting approaches in automatic text retrieval.
Information processing and management, 24(5), 513–523 (1988)
3. Le, Q., Mikolov, T.: Distributed representations of sentences and documents. In
International Conference on Machine Learning, 1188–1196 (2014, January)
4. Mikolov, T., Sutskever, I., Chen, K., Corrado, G. S., Dean, J.: Distributed
representations of words and phrases and their compositionality. In Advances inneural information processing systems 3111–3119 (2013)
5. Elasticsearch, https://www.elastic.co/products/elasticsearch. Last accessed 3
Sept 2018
6. Apache Lucene Scoring Algorithm, http://lucene.apache.org/core/index.html,
Last accessed 3 Sept 2018

A

Summary Example
Bradley was a member of the armed forces who fell while
showering on board his ship.
He applied under s. 21(2) of the Pension Act for benefits
relating to upper back and neck injuries caused by theaccident.
The Veterans Review and Appeal Board dismissed the claim
because the accident was not sufficiently connected withmilitary service as required by s. 21(2).
Bradley made a new claim related to low back pain causedby the same accident.
The Minister of Veterans Affairs indicated that, because
Bradley’s condition resulted from an accident that had
previously been determined as not directly connected toservice, she had no jurisdiction under s. 85(1) of thePension Act to proceed with the application.
Bradley sought judicial review, requesting an order of
mandamus to compel the Minister to determine theapplication.

B

Facts Example[1] Phelan, J.: This Applicant, a self-represented former
member of the Armed Forces, has been locked in litigation
over his disability claim for a slip in the shower of HMCS
Qu’Appelle for several years and in this Court since 1999
(see Bradley v. Canada (Attorney General), [1999] F.C.J.
No. 144). His first application for a pension based on upperback and neck injuries was finally settled in 2004.
[2] After the 2004 Federal Court decision, the Applicant
started a second application for mechanical lower back pain
based on the same incident. After further proceedings it wasdetermined that the Department of Veterans Affairs was
required to determine this new claim. It is the new claim
which is the subject matter of this judicial review.
[3] The Applicant was undergoing officer training as an Acting
Sub-Lieutenant aboard HMCS Qu’Appelle, a destroyer escort. The
ship was alongside in Vancouver near the end of a training
cruise in the Pacific. The ship was ultimately destined to its
home port of CFB Esquimalt.
[4] The Applicant, having completed training for the day, went
to the mess (presumably the Wardroom) where he had a few
beers. The exact quantity was not established but the drinking
took place on board a warship in a tightly regulated
environment.
[5] Mr. Bradley contends that he slipped while in the shower
cleaning up from his daily duties and prior to going ashore.
He was found the next day in his bunk in severe pain. There
was a notation in the medical file that he had significant
alcohol in his system. There was some suggestion drawn that
there was a connection between the shower incident and theconsumption of alcohol on board; however, how that could have
happened in the circumstances of an officer in training, in awardroom, was not established.
[6] When the Department finally considered this new claim, it
reached the conclusion that the fall in the shower did notarise out of, and was not directly connected with, theApplicant’s military service. The Department made no further
determination as to whether the fall caused the mechanical lowback pain.
[7] A Review Panel affirmed the Department’s decision to deny
the Applicant pension entitlement.
[8] The Applicant appealed to the Appeal Board; however, on
August 5, 2008, the Appeal Board affirmed the negativedecision. It is this decision which is the subject of thisjudicial review.