Legal Information Retrieval by Association Rules
Ying Chen1, Yilu Zhou2, Zhen Lu2, Hao Sun2, Wenjun Yang3
2

1
PayPal Inc.
Gabelli School of Business, Fordham University
3
LawCubes Technology Co. Ltd
yzhou62@fordham.edu

Abstract. With the explosive growth of legal documents, automatic Legal Information Retrieval(LIR) is critical to help law professionals to conduct quality services. In this paper, we propose using association rules to retrieval relevant legal
documents. Our method is tested on three COLIEE 2018 tasks, and achieve acceptable and stable performance on all the tasks.
Keywords: Legal Information Retrieval (LIR), Association Rule, n gram, Machine Learning, Random Forest, COLIEE Competition.

1

Introduction

With the explosive growth of legal documents, automatic Legal Information Retrieval(LIR) is critical to help law professionals to conduct quality services. Due to longand complicate syntax structure of legal expressions, high abstract jargons, and limited
datasets, deep learning based NLP approaches which are popular in general domains[15,16,17], perform poorly in legal domain. In this work, we propose an associationrule based framework for legal information retrieval. The contributions of our models:• No supervised training is needed. Therefore, we can score and rank large
amount of legal documents without manual labeling.
• The computational cost is low. The complexity to calculate association scoreis only O(n), make it practical to be used on online legal document searchengines.
• With tailed optimization, the LIR performance of our method is good, which
we will demonstrate using experiments on Competition on Legal Information
Extraction/Entailment (COLIEE 2018) datasets.
This paper is structured as follows: In Section 2 we review the relevant work andpropose our research questions in section 3. We propose our methodology in Section 4.
In Section 5, we describe our experimental design, results, and discussions. We conclude the paper by summarizing our findings and contributions in Section 6.

2

2

Related work

Machine learning based NLP methods are still very popular in LIR researches. [2,3,5,
6, 7,8] use the Support Vector Machine (SVM) and Random Forests (RF) as classifiers,
adopt semantic and syntax features, such as cosine similarity, n-gram intersection, various distance features, dependency bigram, negation similarity, paragraph links, paragraph link types, and etc. [6,9,10, 13] add lexical features such as TF-IDF score intomachine learning classifiers to improve LIR performance. In addition, there are varies
of works experiment on non-machine-learning methods in LIR domain. To automatically retrieval relevant articles for a given legal query, [7] uses K-means clustering,then calculates cosine similarity between queries and articles only in the same cluster.
[1] calculates document similarity by BM25[14] with keyword expansion.
Researchers also ensemble different NLP methods to further improve retrieval efficiency. [4,9,12] ensemble similarity using least square method (LSM ensemble) andlinear discriminant analysis (LDA ensemble). They also add Word2Vec and Latent semantic analysis (LSA) to further boost the overall accuracy. [11] proposes using a 2stage method for LIR: 1) relevance analysis and 2) relevance disambiguation. In therelevance analysis stage, they calculate n-gram association index for each article. For
the ones which have close or low relevance scores, the researchers further leverage termorder probabilities for disambiguation.
We experiment both machine learning based method and association based method
on LIR problems. The best performance is generated by association rule based method
similar to [11], and we further enhance the accuracy. We believe our findings uncovervaluable insights for future researches.

3

Legal Information Retrieval tasks—COLIEE

We research LIR methods within the context of the 5th Competition on Legal Information Extraction and Entailment (COLIEE-2018), which include tasks on both statutelaw and case law. We participant in three tasks in COLIEE 2018 competition: tasks 1,
task 2, and task 3.
• Task 1 is a legal case retrieval task, given a new case 𝑑" , require to retrieve
relevant cases {𝑑#$ , 𝑑#% , ..., 𝑑#& } from a provided case law corpus.
• Task 2 is the legal case entailment task, given a case 𝑑" and a relevant case𝑑# , require identification of paragraphs in 𝑑" that entails the decision of case𝑑" .
• Task 3 is a statute law retrieval task, given a legal bar exam question 𝑑" , require to retrieve relevant statutes {𝑑#$ , 𝑑#% , ..., 𝑑#& } from a provided databaseof civil code statutes.
Three tasks have similar structures, from which we derives the research question forthis paper: given a text 𝑑" , and a training dataset of (𝑑" , 𝑑# ) pairs, searching for relevant
text {𝑑#$ , 𝑑#% , ..., 𝑑#& }.

3

4

Proposed Approach

Since recent studies on LIR mostly leverage machine learning and linguistic similarity,
we propose two sets of modeling frameworks to tackle our research question.
4.1

Machine Learning Based Model adopting Word2Vec/Doc2Vec as features(later refer as RF+Word2Vec model)

Machine learning based models are widely experimented in LIR domain. The mostcommonly used classifiers are Support Vector Machine (SVM) and Random Forest(RF). There are two challenges when using machine learning based methods on LIR
problems: very limited training samples, and long/complex sentences (i.e., large
amount of features). These two challenges make tree-based models be favored overother classifiers such as SVM and neural network(NN), because tree-based models only
choose useful features while SVM and NN have to generate weights for all the features,the latter are much easier to cause overfitting. Therefore, in this paper, we choose to
use RF as our machine learning classifier.
RF Tag Generation
Since LIR is not a natural regression/classification problem, we have to reconstruct
the dataset before feeding it to the classifier. Given a document set {𝑑$ , 𝑑% , … , 𝑑& } anda relevancy pair set 𝑃 = .(𝑑" , 𝑑# )1. Denote all the source document 𝑑" ∈ 𝑄, 𝑄 ={𝑑" , 𝑑" , … }. Denote all the target document 𝑑# ∈ 𝑅, 𝑅 = {𝑑# $ , 𝑑# % , … }. We con$%struct the training dataset 𝑇6 in 2-steps. We first cross-join all the elements in 𝑄 with all
the elements in 𝑅, and remove all the records with 𝑑7 = 𝑑8 (𝑑7 ∈ 𝑄, 𝑑8 ∈ 𝑅). We assign
tag 1 to those (𝑑7 , 𝑑8 ) pairs which can be found in the given relevancy pairs 𝑃, andassign tag 0 for all other records, then dataset 𝑇 is generated:
1 :𝑑7 , 𝑑8 < ∈ 𝑃
𝑇 = {𝑇9 = :𝑑7 , 𝑑8 , 𝑙9 <| 𝑑7 ∈ 𝑄, 𝑑8 ∈ 𝑅, 𝑑7 ≠ 𝑑8 , 𝑙9 = ?
} --(1)0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Then we balance the dataset by keeping all the tag 1s and randomly down-sample
all the tag 0s for each unique input 𝑑" . The training dataset becomes 𝑇6 ∈ 𝑇 such thatfor ∀𝑑" ∈ 𝑄, the number of set .:𝑑" , 𝑑K , 11 ∈ 𝑇6, 𝑑K ∈ 𝑅1 equals to the number of set.:𝑑" , 𝑑& , 01 ∈ 𝑇6, 𝑑& ∈ 𝑅1.
Below is an example demonstrating the dataset reconstruction process. Given a relevancy pair set 𝑃 = .(𝑑" , 𝑑# )1 = {(apple x, pear), (cat, dog)} | 𝑑" ∈ 𝑄, 𝑑# ∈ 𝑅 . We
first cross-join all the elements in 𝑄 with all the elements in 𝑅, and remove all the records with 𝑑" = 𝑑# , then we have a dataset as Table 1. We assign tag 1 to those (𝑑" , 𝑑# )pairs which can be found in the given relevancy pair set 𝑃, and assign tag 0 for all otherrecords. Then we balance the dataset by keeping all the tag 1s and randomly downsample all the tag 0 for each unique input 𝑑" . The balanced dataset is as Table 2. We
then generate Word2Vec/Doc2Vec features for all the records in the balanced dataset
𝑇6.

4
Table 1. Reconstruct dataset 𝑇.
Tag
𝑄
𝑅
apple xpear
1
apple xdog
0
apple xcat
0
pear
apple x
1
pear
dog
0
pear
cat
0
dog
apple x
0
dog
pear
0
dog
cat
1
cat
apple x
0
cat
pear
0
cat
dog
1

W.
Table 2. Balance training dataset 𝑇
tag
𝑄𝑅
apple xpear
1
apple xdog
0pear
apple x
1
pear
dog
0
dog
apple x
0
dog
cat
1
cat
pear
0
cat
dog
1

à

Word2Vec Features
Here is an example demonstrating how Word2Vec features are generated. Assuming
there are two sentences 𝑑"$ and 𝑑"% , unique numbers are used to represent each word.
𝑑"$ : “How is it going?” à Vector 𝑣Z[\ : <1,2,3,4,5>.
𝑑#$ : “I do not know how” à Vector 𝑣Z]\ : <6,7,8,9,1>.
The two vectors share a same element 1 because the original sentences share a sameword “how”. A Word2Vec dictionary can be constructed as follows:
WORD
VECTOR

How
1

is
2

it
3

going
4

?
5

I
6

do
7

not
8

know
9

In addition, we lemmatize the corpus in the preprocessing stage. For example, “am”
“is” “are” are transferred to the word “be”, and represented by a same vector. Python
package Gensim is used to generate Word2Vec features. At the end of the process, eachdocument is converted into an equal length vector (𝑉_`&abc = 𝑁). When record length
shorter than N, the vector will contain (𝑁 − 𝑉_`&abc ) 0s at the end. When record length
longer than N, the words after position N will be omitted. The length we pick is 500,
which is longer than 98.4% of the records. The generated vector is our feature set.
Doc2Vec Features
Here is an example demonstrating how Doc2Vec features are generated. Assuming
there are two documents 𝑑"$ and 𝑑"% , close numbers are used to represent similarwords.
𝑑"$ : “He likes dog” à Vector 𝑣Z[\ : <-0.99,-0.51,0.71>.
𝑑#$ : “She loves cat” à Vector 𝑣Z]\ : <-0.98,-0.53,0.72>.
Python package Gensim is used to generate Doc2Vec features. Users can either train
their own models to define similar words, or rely on system-default movie model todefine similar words. After words being converted to numbers, Singular Value Decomposition (SVD) is used to compress vector dimension to requested length. In the end,

5

the vector length does not represent word numbers, but represent the number of topics
within each document.
With tags and features, the dataset 𝑇6 can be directly used on RF classifiers. We use
RF classifiers in TensorFlow to train. When scoring, we generate full cross-join dataset
𝑇 on the test data without re-balancing. For each 𝑑" ∈ 𝑄, the model prediction scoresof (𝑑" , 𝑑# ) are sorted in descending order, only top K 𝑑# are output as relevant texts.
4.2

Association Rule Model

Machine learning methods have several disadvantages on LIR problems. Firstly, LIR
problem usually have very limited training samples, which make machine learningmodels hard to converge. Secondly, when the sizes of 𝑄 and 𝑅 expand, the space consumption of datasets and the training computational cost exponentially increase. To
enhance the scalability of LIR solutions, we propose following association rule models.
Basic Association Rule Model (later refer as Basic-AR model)
BLEU (bilingual evaluation understudy)[18] algorithm evaluates the quality of machine-translation which transforms one natural language to another. We borrow theconcept of BLEU score and propose association rule model to search for relevant texts.
Given a document set {𝑑$ , 𝑑% , … , 𝑑& }, we transform 𝑑" into n_gram set 𝑛Z[ , transform 𝑑# into n_gram set 𝑛Z] . Then the association score of (𝑑" , 𝑑# ) can be calculatedusing the following equation:𝑠Z[g Z]h =i&j ∩&j i[g]hi&j ∪&j i[g]h=i&j ∩&j i[g]hi&j imi&j ini&j ∩&j i
[g
]h
[g
]h

--(2)

where i𝑛Z[g ∩ 𝑛Z]h i is the number of n_grams within the insection of 𝑛Z[g and 𝑛Z]h ,and i𝑛Z[g ∪ 𝑛Z]h i is the number of n_grams within the union of 𝑛Z[g and 𝑛Z]h ,i𝑛Z[g ∪ 𝑛Z]h i = i𝑛Z[g i + i𝑛Z]h i − i𝑛Z[g ∩ 𝑛Z]h i . For each 𝑑" ∈ 𝑄, we calculate its association score against all the 𝑑# ∈ 𝑅, then sort the association scores of (𝑑" , 𝑑# ) indescending order, only top K 𝑑# are output as relevant texts.
Below is an example demonstrating how to calculate associate score of (𝑑" , 𝑑# ).
𝑑"$ : “a cat is on the mat.” à bigram set 𝑛Z[\ : {a cat, cat is, is on, on the, the mat}.
𝑑"% : “i have a cat in my home.” à bigram set 𝑛Z[p : {i have, have a, a cat, cat in, in
my, my home}.
𝑑#$ : “there is a cat on the mat.” à bigram set 𝑛Z]\ : {there is, is a, a cat, cat on, onthe, the mat}.
𝑑#% : “a cat is in my home.” à bigram set 𝑛Z]p : {a cat, cat is, is in, in my, my home}.
The associate score of each (𝑑" , 𝑑# ) should be as Table 3. According to Table 3, fordocument 𝑑"$ , document 𝑑#$ is more relevant than document 𝑑#% , while for document𝑑"% , document 𝑑#% is more relevant than document 𝑑#$ .

6Table 3. Association score example.
𝑠
𝑑#$
𝑑#%
𝑑"$
3/8
2/8
𝑑"%
1/11
3/8

Supervised: Co-occurrence Association Rule Model (later refer as CoOccur-AR
model)
Basic association rule model only considers the similarity between the source document
and the target document, it does not leverage human-labeled relevancy dictionaries. To
find out whether adding knowledge-based information helps or not on LIR problems,
we propose an alternative association rule model.
Given a document set {𝑑$ , 𝑑% , … , 𝑑& } and a relevancy pair set 𝑃 = .:𝑑" , 𝑑# <1, wetransform 𝑑" into n_gram set 𝑛Z[ , and 𝑑# into n_gram set 𝑛Z] . Define an n_gram pair
set between each n_gram 𝑤9 within set 𝑛Z[ and each n_gram 𝑤_ within set 𝑛Z] to be:
𝑁𝑃"# = {(𝑤9 , 𝑤_ ) | 𝑤9 ∈ 𝑛Z[ , 𝑤_ ∈ 𝑛Z] }. Denote 𝑁𝑁 to be the set which contains all
the n_grams within 𝑛Z[ and 𝑛Z] : 𝑁𝑁 = ⋃Zs∈t∪u 𝑛Zs . For ∀𝑤9 ∈ 𝑁𝑁, ∀𝑤_ ∈ 𝑁𝑁, weconstruct a co-occurrence association matrix which count the frequency of (𝑤9 , 𝑤_ )appear as a pair in the relevancy pair set 𝑃. We also record the term frequency as #𝑑9
when 𝑛9 = 𝑛_ .
#{𝑁𝑃"# | (𝑤9 , 𝑤_ ) ∈ 𝑁𝑃"# | (𝑤_ , 𝑤9 ) ∈ 𝑁𝑃"# } 𝑛9 ≠ 𝑛_
𝑀(𝑤9 , 𝑤_ ) = ?
--(3)#{𝑑9 | 𝑤9 ∈ 𝑑9 , (𝑑9 ∈ 𝑄 | 𝑑9 ∈ 𝑅)}
𝑛9 = 𝑛_Finally, we normalize the co-occurrence association matrix by dividing its term frequency.
W (𝑤9 , 𝑤_ ) = x(ys,yz )𝑀--(4)x(ys ,ys )W (𝑤9 , 𝑤_ ) | 𝑤9 ∈ 𝑛Z , 𝑤_ ∈For each 𝑑" ∈ 𝑄, we calculate its association score (∑ 𝑀[𝑛Z] ) against all the 𝑑# ∈ 𝑅, then sort the association scores of 𝑑# in descending order,
only top K 𝑑# are output as relevant texts.
Below is an example demonstrating how to construct co-occurrence associate matrix
and calculate co-occurrence associate score for (𝑑" , 𝑑# ). Assume the relevancy pair set
𝑃 = .:𝑑" , 𝑑# <1 ={(apple x, pear), (cat, dog), (apple y, dog), (apple x, pear)} | 𝑑" ∈ 𝑄, 𝑑# ∈ 𝑅 .
If we only consider unigram, we can count the number of times when each unigram
𝑤9 appears together with each unigram 𝑤_ . As to the diagonal of the matrix, we records
the number of times each unigram appears in the relevancy pair set 𝑃. The co-occurrence association matrix 𝑀 should be as Table 4.
Table 4. Co-occurrence association matrix 𝑀.
Co-occurrence
apple
pear
cat
dog
apple
3
2
1
pear
2
2
cat
1
1
dog
1
1
2
x
2
y
1

x

y

2
1
2
1

7

Then we normalize the co-occurrence association matrix, dividing each row by its
diagonal value (i.e., number of times each unigram appears in 𝑃). The purpose for normalization is to remove the effect of unbalanced term frequency in the texts.
W.
Table 5. Normalized co-occurrence association matrix 𝑀
Normalized Co-occurrence
apple
pear
cat
dog
apple
1
2/3
1/3
pear
1
1
cat
1
1
dog
1/2
1/2
1
x
1
y
1

x

y

1
1/2
1
1

Then for the document "Apple x" ∈ 𝑄, we calculate its co-occurrence association
W (𝑤9 , 𝑤_ ) | 𝑤9 ∈ 𝑛Z , 𝑤_ ∈ 𝑛Z ) against all the 𝑑# ∈ 𝑅:
score (∑ 𝑀[]
(apple x, pear): 5/3
Co-occurrenceapplex(apple x, cat): 0
Co-occurrence
apple
x

pear
2/3
1
cat

(apple x, dog): 1/3
Co-occurrence
apple
x

dog
1/3

(apple x, apple y): 1
Co-occurrence
apple
x

apple
1

y

According to the co-occurrence association scores, the relevancy of (apple x, pear)>(apple x, apple y)>(apple x, dog)>(apple x, cat).

5

Experiments and Results

We experiment our LIR models on three tasks within COLIEE 2018 competition: tasks
1, task 2, and task 3.
5.1

Task 1: The Legal Case Retrieval Task

Experimental Setup
Task 1 is a legal case retrieval task, given a new case 𝑑" , require to retrieve relevantcases {𝑑#$ , 𝑑#% , ..., 𝑑#& } from a provided case law corpus.
In this task, the size of 𝑄 is 286, the size of 𝑅 is 200. The relevancy pair 𝑃 is also
given, but for each document 𝑑" ∈ 𝑄, the number of its relevant cases specified in 𝑃
varies from 1 to 53, as shown in Fig 1. We remove common English stop words andfeed the documents into our LIR models for experiments.
Parameter Adjustment•

For the RF-Word2Vec model, we reconstruct the balanced training dataset,then use Gensim Doc2Vec [19] to generate 50 numbers long vectors from (𝑄,8•𝑅) for each record. 30% of the training dataset is reserved for validation. Thebalanced training dataset includes 5628 records, tested depths of random forest
include [5, 20, 30, 50, 100], 50 performs best on the validation dataset, andtree_number=100.
For the Basic-AR model, we tried {1~4}_gram, and find bigram performs bestin this case. we further calculate document frequency (DF) for each term 𝑡 as
follows:$&•K‚`# ƒ„ b7K`… b †‡‡`†# 7& Zs
𝐷𝐹b = ∑9
--(5)
9

•

bƒb†_ &•K‚`# ƒ„ b`#K… 7& Zs

𝑑9 is the kth document. We remove the terms with document frequency
larger than 0.02, for those frequently appear terms can be consider as stop
words. However, we do not set the lower bound of document frequency, because legal terms are very domain-dependent, un-common terms can havegreat prediction power in finding relevant legal documents.
In addition, we ensemble the RF-Word2Vec and the Basic-AR models, andtest its prediction power. To be specific, we treat the AR_score as a feature,and feed it into the RF-Word2Vec model for classification. We later refer thisexperimental setting as RF-Word2Vec+AR.

Fig 1. Number of relevant cases foreach case in Task 1.

Table 7. Task 1: Basic-AR performance on test
datasetF-measPreciReid
ure
sion
call
HUKB1
0.38
0.50
0.31
HUKB2
0.35
0.40
0.30
JNLP-r=2.5
0.60
0.55
0.66
JNLP-k=10
0.65
0.68
0.63
Smartlaw
0.34
0.29
0.43
UA
0.35
0.37
0.32
UA-postproc
0.37
0.35
0.40
UA-smote
0.37
0.35
0.39
UBIRLED-1
0.22
0.13
0.62
UBIRLED-2
0.31
0.20
0.72
UBIRLED-3
0.17
0.56
0.10
UL
0.39
0.56
0.30

Evaluation Method
We adopt the evaluation method of COLIEE Task 1 to measure the performance of our
LIR models.
=(the number of correctly retrieved cases(paragraphs) for all queries)(the number of retrieved cases(paragraphs) for all queries)

𝑅𝑒𝑐𝑎𝑙𝑙 =(the number of correctly retrieved cases(paragraphs) for all queries)(the number of relevant cases(paragraphs) for all queries)

𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛

𝐹 − 𝑚𝑒𝑎𝑠𝑢𝑟𝑒 =(% – —˜™š›œ›•ž – Ÿ™š ¡¡)(—˜™š›œ›•ž m Ÿ™š ¡¡)

--(6)

9

Results
On the validation dataset, Basic-AR Model performs best among the three models. The
best f-measure is achieved when output top 9 cases as relevant cases.

Top
n
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

Table 6. Task 1: Basic-AR performance on validation dataset
AR
RF
RF+AR
F-meas- precireF-meas- preciF-measure precision
recallure
sion
call ure
sion
0.17
0.54
0.1
0
0.02
0
0
0.04
0.25
0.5
0.16
0.01
0.02
0.01
0.03
0.04
0.3
0.47
0.22
0.01
0.03
0.01
0.03
0.06
0.33
0.44
0.26
0.02
0.02
0.01
0.03
0.04
0.35
0.42
0.3
0.02
0.03
0.02
0.05
0.06
0.37
0.4
0.34
0.02
0.03
0.02
0.05
0.06
0.37
0.38
0.37
0.02
0.03
0.02
0.05
0.06
0.38
0.36
0.39
0.02
0.02
0.02
0.04
0.04
0.38
0.35
0.42
0.02
0.02
0.02
0.04
0.04
0.38
0.34
0.44
0.02
0.02
0.02
0.04
0.04
0.38
0.33
0.46
0.02
0.02
0.03
0.05
0.04
0.38
0.32
0.47
0.02
0.02
0.03
0.05
0.04
0.37
0.3
0.49
0.03
0.02
0.03
0.05
0.04
0.37
0.29
0.51
0.03
0.02
0.03
0.05
0.04
0.37
0.28
0.52
0.03
0.02
0.03
0.05
0.04
0.36
0.28
0.53
0.03
0.02
0.03
0.05
0.04
0.36
0.26
0.54
0.03
0.02
0.04
0.05
0.04
0.35
0.26
0.55
0.03
0.02
0.04
0.05
0.04

The Basic-AR model performance holds well on test dataset (Table 7). And our
method is with low cost, scalable, and have stable performance over different datasets.

5.2

Task 2: The Legal Case Entailment task

Experimental Setup
Task 2 is the legal case entailment task, given a case 𝑑" and a relevant case 𝑑# , require
identification of paragraphs in 𝑑" that entails the decision of case 𝑑" .
In this task, the size of 𝑄 is 181. The relevancy pair 𝑃 is also given, but for eachdocument 𝑑" ∈ 𝑄, the number of its relevant paragraphs specified in 𝑃 varies from 1 to4, as shown in Fig 4.
Parameter Adjustment•

For the RF-Word2Vec model, we reconstruct the balanced training dataset,then use Gensim Doc2Vec [19] to generate 50 numbers long vectors of (𝑄, 𝑅)
for each record. 30% of the training dataset is reserved for validation. Thebalanced training dataset includes 478 records, the tested depths of random

recall
0
0.02
0.02
0.02
0.04
0.04
0.04
0.04
0.04
0.04
0.06
0.06
0.06
0.06
0.06
0.06
0.08
0.08

10

•

•

forest include [5, 20, 30, 50, 100], 20 performs best on the validation dataset,and tree_number=100.
For the Basic-AR model, we tried {1~4}_gram, and find bigram performs bestin this case. In this task, We do not remove the terms with high document
frequency, because after removing terms with high document frequency, prediction performance drops. Similar to task 1, we do not set the lower boundof document frequency.
In addition, we ensemble the RF-Word2Vec and the Basic-AR model, and test
its prediction power.

Fig 4. Number of relevant paragraphs
for each case in Task 2.

Table 9. Task 2: Basic-AR performance on test dataset (ID: Smartlaw)
idF-measure Precision Recall
Smartlaw
0.07
0.05
0.15
UA
0.26
0.24
0.28
UA100estimators
0.21
0.19
0.23
UA500estimators
0.26
0.24
0.28
UBIRLED-1
0.09
0.05
0.83
UBIRLED-2
0.09
0.05
0.92
UBIRLED-3
0.09
0.05
0.79
UNCC0
0.04
0.03
0.06

Evaluation Method
Model performance on task 2 is evaluated using the same F-measure as task 1, which
listed in equation (6).
Top
n
1
2
3
4

Fmeasure
0.07
0.07
0.09
0.09

Table 8. Task 2: Basic-AR performance on validation dataset
AR
RF
AR+RF
FFprePreci- Repreci- remeasmeas- cirecall
sion
call
sion
call
ure
ure
sion
0.07
0.07
0
0.01
0
0.00
0.02
0.00
0.06
0.11
0.01
0.01
0.02
0.03
0.02
0.04
0.06
0.16
0.02
0.01
0.03
0.03
0.02
0.06
0.06
0.2
0.02
0.01
0.04
0.03
0.02
0.08

Results
On the validation dataset, Basic-AR Model performs best. The best f-measure keepincreasing when K increases. Because in grant truth, all the cases have less than 4 relevant paragraphs, we select top K=4 as relevant paragraphs and do not go beyond 4. The
Basic-AR model performance holds well on the test dataset (Table 8).

11

5.3

Task 3: The Statute Law Retrieval Task

Experimental Setup
Task 3 is a statute law retrieval task, given a legal bar exam question 𝑑" , require to
retrieve relevant statutes {𝑑#$ , 𝑑#% , ..., 𝑑#& } from a provided database of civil code statutes.
In this task, the size of 𝑄 is 657, and the size of 𝑅 is 1056. The relevancy pair 𝑃 is
also given, but for each document 𝑑" ∈ 𝑄, the number of its relevant cases specified in
𝑃 varies from 1 to 7, as shown in Fig 7. We remove common English stop words andfeed the documents into our LIR models for experiments.

Fig 7. Number of relevant articles for each query in Task 3.

Parameter Adjustment•••

For the Basic-AR model, we experiment on {1~4}_gram, and find bigram+trigram performs best in this case. We remove the terms with document frequency larger than 0.01, for those frequent appear terms can be consider asstop words. We do not set the lower bound of document frequency, for uncommon terms can have great prediction power in finding relevant articles.
For the RF-Word2Vec model, we reconstruct the balanced training dataset,then use Gensim Word2Vec to generate 500 numbers long vectors of (𝑄, 𝑅)
for each record. 30% of the training dataset is reserved for validation. Thebalanced training dataset includes 1380 records, the tested depths of random
forest include [5, 20, 30, 50, 100], 30 performs best on the validation dataset,and tree_number=1000. Since RF+AR performs better than RF in both task 1
and task 2, we ensemble the RF-Word2Vec and the Basic-AR model in task
3, to test its prediction power.
For the CoOccur-AR model, we experiment on {1~4}_gram, and find trigram
performs best in this case.

Evaluation Method
Model performance on task 3 is evaluated using a different F2-measure comparing totask 1 and task 2. This evaluation matrix focus more on recall than precision.
𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = average of(the number of correctly retrieved articles for each query)(the number of retrieved articles for each query)12

𝑅𝑒𝑐𝑎𝑙𝑙 = average of(the number of correctly retrieved articles for each query)(the number of relevant articles for each query)
𝐹2 − 𝑚𝑒𝑎𝑠𝑢𝑟𝑒 =(5 x Precision x Recall)(4 x Precision + Recall)− −(7)

Table 10. Task 3: 3 model performance on validation dataset
F2-measure
RecallPrecisionTop
CoCoConAR
RF+AR AR
RF+AR AR
RF+AR
AR
AR
AR
1 0.48 0.23
0.20
0.47 0.23
0.19
0.53 0.26
0.23
2 0.49 0.25
0.20
0.51 0.29
0.20
0.43 0.17
0.18
3 0.49 0.26
0.19
0.54 0.34
0.21
0.36 0.13
0.15
4 0.48 0.24
0.19
0.56 0.36
0.22
0.31 0.11
0.13
5 0.47 0.24
0.19
0.57 0.39
0.23
0.28 0.09
0.12
6 0.46 0.23
0.20
0.58 0.41
0.24
0.25 0.08
0.11
7 0.45 0.21
0.20
0.59 0.42
0.26
0.23 0.07
0.10
Table 11. Task 3: Basic-AR performance on test dataset (ID:smartlaw)
F2 measPrecilang sid
ure
sion
Recall MAP R_5 R_10
E
UB3
0.70
0.78
0.69
0.80
0.80
0.85
J
HUKB2
0.69
0.77
0.68
0.78
0.79
0.84
J
HUKB1
0.68
0.75
0.68
0.78
0.79
0.84
J
ORGJ1
0.66
0.74
0.65
0.77
0.78
0.84
?
YA
0.66
0.72
0.65
0.75
0.73
0.75
E
ORGE1
0.64
0.71
0.63
0.74
0.75
0.81
E
UB2
0.62
0.68
0.62
0.75
0.80
0.87
E
JNLP1
0.61
0.41
0.71
0.74
0.76
0.82
E
smartlaw
0.60
0.41
0.70
0.70
0.71
0.76
E
JNLP2
0.60
0.41
0.70
0.73
0.75
0.81
E
SPABS_bm25
0.58
0.40
0.67
0.71
0.78
0.82
E
UE
0.45
0.49
0.45
E
smartlaw_3gram
0.44
0.49
0.43
0.47
0.45
0.46
E
UB1
0.42
0.45
0.41
0.54
0.57
0.72
?
smartlaw_2gram
0.34
0.30
0.43
0.46
0.44
0.48
E
SPABS_rnnen
0.22
0.14
0.25
0.26
0.34
0.45
E
SPABS_rnnsq
0.20
0.12
0.23
0.27
0.35
0.45

R_30
0.96
0.93
0.93
0.93
0.85
0.90
0.96
0.92
0.83
0.91
0.91
0.51
0.82
0.52
0.57
0.61

Results
On the validation dataset, Basic-AR model performs best among the three models.
Basic-AR and RF-Word2Vec models achieve the best f-measure when output top 2
articles as relevant articles, while CoOccur-AR model achieves the best f-measure
when output top 3 articles as relevant articles. The Basic-AR model performance holds
well on test dataset (Table 11). On the test dataset, Basic-AR model performance iseven better.

13

5.4

Discussion

We experiment our three LIR models on task 1, task 2, and task 3 within COLIEE 2018
competition. The Basic-AR achieve the best performance over RF-Word2Vec and
CoOccur-AR on all the three tasks, which suggests unsupervised and non-machinelearning based method have great potential on LIR problems. In addition, Basic-AR
model is with low cost, scalable, and have stable performance over different datasets,
it can save both human efforts and computational costs on legal information retrieval
tasks.
We have tried several attempts to further tune the performance of Basic-AR. Firstly,
instead of using (single-n) gram, we found (multi-n) gram can generate better performance. In task 3, it has been proved that bigram+trigram achieves the best prediction
performance. Secondly, in this paper, we only try to output static top K 𝑑# as output.
For example, in task 3, we find the optimal K to be 2, then for all the queries 𝑑" , weonly consider top 2 article 𝑑# as relevant articles. However, some queries may have
only 1 relevant article while others may have 3 relevant articles. Static top K outputs
may not satisfy such need. We experiment on using Basic-AR score as threshold forcut off, but the performance is not as good as static top K. Future researches is required
to find a better relevancy cut off.

6

Conclusion

In this paper, we propose three models to tackle LIR problems, and we experiment on
three COLIEE 2018 tasks. The association based model outperform the rest two models,and achieve stable and acceptable performance on all three COLIEE tasks. Therefore,
we demonstrate non-machine learning method has great potential on LIR problems.

References
1. Daiki Onodera and Masaharu Yoshioka, “Civil Code Article Information Retrieval System
based on Legal Terminology and Civil Code Article Structure”, Tenth International Workshop on Juris-informatics (JURISIN), 2016 (Submission ID: HUKB)
2. Truong-Son Nguyen, Viet-Anh Phan, Thanh-Huy Nguyen, Hai-Long Trieu, Ngoc-Phuong
Chau, Trung-Tin Pham, and Le-Minh Nguyen, “Legal Information Extraction/Entailment
Using SVM-Ranking and Tree-based Convolutional Neural Network”, Tenth International
Workshop on Juris-informatics (JURISIN), 2016 (Submission ID: JNLN2)
3. Danilo S. Carvalho, Vu Duc Tran, Khanh Van Tran, Viet Dac Lai, and Minh-Le Nguyen,
Lexical to Discourse-level Corpus Modeling for Legal Question Answering”, TenthInternational Workshop on Juris-informatics (JURISIN), 2016 (Submission ID: JNLN1)
4. Kiyoun Kim, Seongwan Heo, Sungchul Jung, Kihyun Hong and Young-Yik Rhim, “AnEnsemble Based Legal Information Retrieval and Entailment System”, Tenth International
Workshop on Juris-informatics (JURISIN), 2016 (Submission ID: iLis7)
5. Phong-Khac Do, Huy-Tien Nguyen, Chien-Xuan Tran, Minh-Tien Nguyen and Nguyen
Le Minh, “Legal Question Answering using Ranking SVM and Deep Convolutional
Neural Network”, Tenth International Workshop on Juris-informatics (JURISIN), 2016
(Submission ID: JNLN3)

14
6. Mi-Young Kim, Ying Xu, Yao Lu and Randy Goebel, “Legal Question Answering Using
Paraphrasing and Entailment Analysis”, Tenth International Workshop on Jurisinformatics(JURISIN), 2016 (Submission ID: UofA)
7. Adebayo Kolawole John , Luigi Di Caro, Guido Boella, and Cesare Bartolini,“TEAMNORMAS' PARTICIPATION AT THE COLIEE 2016 BAR LEGAL EXAMCOMPETITION”, Tenth International Workshop on Juris-informatics (JURISIN), 2016
(Submission ID: N01)
8. Yoshioka, M., & Onodera, D. (2017). A Civil Code Article Information Retrieval System
based on Phrase Alignment with Article Structure Analysis and Ensemble Approach. In 4thCompetition on Legal Information Extraction and Entailment (COLIEE 2017), 16th International Conference on Artificial Intelligence and Law (ICAIL 2017) .
9. Heo, S., Hong, K., & Rhim, Y.-Y. (2017). Legal Content Fusion for Legal Information Retrieval. In the 16th International Conference on Artificial Intelligence and Law (ICAIL
2017).
10. Nguyen, T.-S., Phan, V.-A., & Nguyen, L.-M. (2017). Recognizing entailments in legal texts
using sentence encoding-based and decomposable attention models. In 4th Competition on
Legal Information Extraction and Entailment (COLIEE 2017), 16th International Conference on Artificial Intelligence and Law (ICAIL 2017) .
11. Carvalho, D. S., Tran, V., Tran, K. Van, & Nguyen, L. M. (2017). Improving Legal Information Retrieval by Distributional Composition with Term Order Probabilities. In 4th Competition on Legal Information Extraction and Entailment (COLIEE 2017), 16th International
Conference on Artificial Intelligence and Law (ICAIL 2017).
12. Nanda, R., John, A. K., Caro, L. Di, Boella, G., & Robaldo, L. (2017). Legal Information
Retrieval Using Topic Clustering and Neural Networks. In 4th Competition on Legal Information Extraction and Entailment (COLIEE 2017), 16th International Conference on Artificial Intelligence and Law (ICAIL 2017) .
13. Kim, M.-Y., & Goebel, R. (2017). Two-step Cascaded Textual Entailment for Legal Bar
Exam Question Answering. In the 16th International Conference on Artificial Intelligenceand Law (ICAIL 2017) .
14. S. E. Robertson and S. Walker. Okapi/Keenbow at TREC-8. In Proceedings of TREC-8,
pages 151{162, 2000.
15. Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying
Song, and Rabab Ward. Deep sentence embedding using long short-term memory networks:
Analysis and application to information retrieval. IEEE/ACM Trans. Audio, Speech andLang. Proc.,24(4):694{707, April 2016.
16. Guido Zuccon, Bevan Koopman, Peter Bruza, and Leif Azzopardi. Integrating and evaluating neural word embeddings in information retrieval. In Proceedings of the 20th Australasian Document Computing Symposium, ADCS '15, pages 12:1{12:8, New York, NY, USA,2015. ACM.
17. Debasis Ganguly, Dwaipayan Roy, Mandar Mitra, and Gareth J.F. Jones. Word embedding
based generalized language model for information retrieval. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval,
SIGIR '15, pages 795-798, New York, NY, USA, 2015. ACM.
18. Papineni, K.; Roukos, S.; Ward, T.; Zhu, W. J. (2002). BLEU: a method for automatic evaluation of machine translation (PDF). ACL-2002: 40th Annual meeting of the Association
for Computational Linguistics. pp. 311–318. CiteSeerX 10.1.1.19.9416 .
19. Rathore, M. (2018). gensim doc2vec & IMDB Sentiment Dataset. Retrieved fromhttps://markroxor.github.io/gensim/static/notebooks/doc2vec-IMDB.html