JNLP Group: Legal Information Retrieval with
Summary and Logical Structure Analysis
Vu Duc Tran, Son Truong Nguyen, and Minh Le Nguyen
School of Information Science,
Japan Advanced Institute of Science and Technology (JAIST),
Nomi, Ishikawa, Japan.
{vu.tran, nguyen.son, nguyenml}@jaist.ac.jp

Abstract. We present our methods for solving the legal case retrieval
and statute law retrieval tasks in Competition on Legal Information Extraction/Entailment, 2018. We explore the benefits from analyzing legal
documents’ summaries and logical structures to tackle these legal information retrieval tasks.
Keywords: legal texts, deep learning, document representation, structure analysis, information retrieval

1

Introduction

Automatic legal document processing systems can speed up significantly thework of experts, which, otherwise, requires significant time and efforts. One
crucial kind of such systems, automatic information retrieval whose systems,
in place of experts, process over enormous amount of documents, for example,
legal case reports and statute law documents, which are accumulated rapidly
over time.
In this paper, we describe our method for tackling the legal case retrieval
and statute law retrieval tasks in Competition on Legal Information Extraction/Entailment, 2018. These are the main points of our work:– Legal case retrieval task
• We developed an approach to obtain document representation in continuous vector space in which the summary properties of the document areembedded, which we call encoded summarization.
• We extracted and compared lexical features on different parts of queries
and candidates including: summary, fact, and paragraphs.
• We found that combining the lexical features and the encoded summarization achieved better results than only each of them.
– Statute law retrieval task
• We improved the system in [10] that allow to use the result of legal
structure analysis as an additional information for measure the relevance
between a query and an article.

• We found that sometimes that additional information can improve theperformance of the task.
The remainder of this paper are as follows. Section 2 describes our proposedmodels, experiments and results for the legal case retrieval task. Section 3 describes the proposed models, experiments and results for the statute law retrieval
task. Finally, Section 4 concludes our work.

2

Legal Case Retrieval Task

2.1

Task Overview

The legal case retrieval task involves reading a new case q, and extracting supporting cases c∗1 , c∗2 , ..., c∗n for the decision of q from a given list of candidate
cases. The candidate cases that support for the decision of a new case are called’noticed cases’. The legal cases are sampled from a database of predominantly
Federal Court of Canada case laws, provided by Compass Law.
When dealing with this task, we observed several obstacles. First, the candidate cases are 5.7K-token long in average. This poses the problem of understanding the reason of selecting the cases as supporting cases. We, then, choseanother approach which is comparing the summaries of each query and its candidate cases. We, however, found that the summary of the query is not necessarily
lexically similar to the summary of the candidate cases. Moreover, some candidate cases do not have summary at all. We would like to obtain the summary foreach and every candidate cases, and furthermore, the summary should be comparable with the summary of the corresponding query. We come up with the ideaof encoding the entire document into vector space embedding the properties ofsummarization, and we called it encoded summarization. Thus, we developed a
combination method utilizing the lexical matching and encoded summarization.
2.2

Lexical Matching

We estimate the lexical matching between a query and a candidate case in different types of n-grams, skip-grams, longest common subsequence to measurevarious degrees of lexical similarity. We employ the following matching formulasfrom ROUGE evaluation[5]:– ROUGE-1: Unigram matching. This measures word overlapping between aquery and a candidate case.
– ROUGE-2: Bigram matching. Similar to ROUGE-1 with word replaced by
two consecutive words as one token.
– ROUGE-L: Longest common subsequence. This measures the strictly ordered overlapping scattering over the texts.
– ROUGE-W: Weighted longest common subsequence. This favors matcheswith subsequences having less distance between words. While ROUGE-L
can provide the information about strictly ordered overlapping scattering
around, ROUGE-W can provide more precise overlapping, for example, consecutive matches.

– ROUGE-S: Skip-Bigram. This measures the co-occurrence of all pair of words
in their sentence order.
– ROUGE-SU: Skipped bi-gram + uni-gram. This is a kind of combination of
ROUGE-1 and ROUGE-S, the tokens are skip-bigrams and unigrams.
To have more precise comparison between a query and a candidate, we compute 4 matching options:– Summary-Summary: matching of the query’s summary with the candidate’ssummary. This matching represents the comparison of the highlights between
the query and the candidate.
– Fact-Summary: matching of the query’s fact with the candidate’s summary.
This matching represents the ratio of the candidate summary mentioning
relevant fact.
– Summary-Paragraphs: matching of the query’s summary with the candidate’s paragraphs. This matching represents the ratio of the query’s highlights mentioned in the candidate’s detail.
– Fact-Paragraphs: matching of the query’s fact with the candidate’s paragraphs. This matching represents the ratio of the query’s fact also occurredin the candidate’s detail.
We compute precision, recall, and f-measure of the above matching operatorsand matching options as lexical matching features, where each of these metrics
represents different aspects of the matching between a source and a target. For
example, with the option of fact-paragraphs as source-target, the precision measures how much of the fact is mentioned in the paragraphs, the recall measures
how much content of the paragraphs is mentioned in the fact, and the f-measure
is a balance measurement of both precision and recall.
2.3

Encoded Summarization

Summary contains the highlights of a document, which is an important factor in
relevance analysis. As observed from the dataset, the query’s summary may notsimilar to the relevant cases’ summaries. We would like to extend the summary of
both the query and the candidates to include more points from fact/paragraphs.
In [11], the authors present a way of obtaining catchphrases of legal casedocuments via phrase scoring using deep neural networks. Catchphrases represent important legal points of a legal case document and are usually draftedby experts, thus play important roles for understanding the case. To generate
catchphrases, they built a scoring model to estimate phrasal scores of a legal casedocument and used the phrasal scores to construct predicted catchphrases for a
new legal case. The scoring model is trained given expert drafted catchphrases ofthe document and optimized with the objective that catchphrases are expected
to have higher scores than other contents in the document. In other words, we
can train such a scoring model given training documents with indicated important contents which are document summaries in our task.

Inspired from the work’s learning process, we propose to obtain document
embedding directed by document summary. We estimate the phrasal scores foreach document given its summary (extracted using indicators ‘Summary:’ and‘Present:’) and paragraphs. We then train the scoring model with the objectivethat summary contents are expected to have higher scores than paragraph contents. After obtaining the trained model, instead of following the second phase
to select catchphrases, we generate encoded summary in continuous vector spaceby composing document vectors from latent representations of the model. The
encoded summary represents the summarization nature of the document.
Phrase Scoring Model We describe the phrase scoring model of [11] for thistask.
Given a document, we denote wjsi as word j th of sentence ith . The featuresof word wj of a sentence are captured using convolutional neural layer [2–4, 9]
as follows:
v(wj−l )




 ...

 c

 v(wj ) 
fwj = ReLU 
W
(1)







 ...

v(wj+l )where, v(·) : 7→ Rd : word embedding vector lookup map, l: corresponding to thewindow size containing 2l + 1 words, [·] ∈ Rdl : concatenated embedding vector,
Wc ∈ Rc×dl : convolution kernel matrix with c filters, fwj ∈ Rc : word feature
vector, ReLU : rectified linear unit activation.
Sentence features fsi are, then, captured by applying max pooling over thewhole sentence.
fsi = max-poolingj (fwsi )j(2)

where max-pooling are operated over each dimension of vectors fwsi .
jWith the same max-pooling operation as above, we compute document features as:
fd = max-poolingi (fsi )(3)

Finally, we apply a multilayer perceptron (MLP) with one hidden and one
output layer
M LP (x) = sigmoid(W2 · tanh(W1 · x + b1 ) + b2 )(4)

to compute the score of each word wjsi as
f w sij
P wjsi , si , d = M LP  fsi fd(5)where the hidden layer computes the word representative features respecting to
its local use (surrounding neighbors), its enclosing sentence, and its document.
The word representative features are fed to the output layer to compute wordscore (ranging from 0.0 to 1.0). The model is trained following the optimization
process in [11] with summary acting as gold catchphrases and paragraphs acting
as document sentences.
Document Vector Composition We present our method of composing document vectors from the phrase scoring model.
First, given a document, we obtain its the phrase scores and its internal
representations: phrase level, sentence level and document level encodings.
Then, we compose the document vector as:
i
 h
P
si
si
w
,
s
,
d
×
f
;
f
;
f
P
i
d
s
i
wj
j
i,j

P
(6)
g(d) =
si
P
w
,
s
,
d
i
j
i,j
This composition resembles summarization where we weight the document
internal representations by its summary. Thus, we call this composition encodedsummarization.
Query-Candidate Relevance Vector Each dimension in one document vector is obtained from composition of the same kernel representing one feature.
By comparing each dimension independently, we can estimate the compatibilityof a query and a candidate over the dimension. Thus, we compute the querycandidate relevance vector as the element-wise product of query vector and candidate vector.
First, we obtain query vector g(q) and candidate vector g(c) using the document vector composition in Equation 6. Then, we compute the relevance vectorof query q and candidate c by the following element-wise product.
h(q, c) = g(q)
2.4

g(c)

(7)

Learning to Rank Candidates

We formulate the task as bipartite ranking problem and devise the learning toranking method to solve it using lexical matching features from Sections 2.2 andquery-candidate relevance features from Section 2.3.
We utilize pair-wise ranking strategy: pairing each noticed case with an irrelevant case from the candidate list.
We adopt Linear-SVM as the learning algorithm for solving the optimization
problem.
After obtaining the scored candidates as a ranked list, we proceed to selecttop candidates as the predicted noticed cases. We use two selection strategies:
1) based on hard top k, 2) based on the score range from top 1 candidate and isbound relatively by score deviation.

– Top k: selecting top k highest scored candidates.
– Range r: selecting all cases whose scores in the interval:
Range(r) = (top 1 score − r × mean score standard deviation, top 1 score](8)
where mean score standard deviation is the mean of score standard deviations of the supporting cases of the training queries and is calculated in
Equation 9. By applying a fixed score range over the ranked candidate cases,
we only select candidate cases whose score differences among them do not
exceed the range which is not achievable with the previous strategy.
mean score standard deviation =

2.5

1 Xp
V ar[c∗ ]
|q| q

(9)

Experiments

We evaluated our approach by performing leave-one-out validation. From theorganizer provided 285 training queries, we used each and every one query as
validation data and the other 284 queries as training data, which creates 285
folds for leave-one-out validation.
Table 1. Phrase scoring model parameters.
Parameter

Description

Embeddings (vector size d)
GloVe [8] d = 300
300
CNN filters c
CNN window size 2l + 1
5
MLP hidden size
300
Optimizer
Adam[1]0.0001
Learning rate
Gradient clipping max norm
5.0
Loss coefficients (a1 , a2 , b1 , b2 , b3 , b4 ) (1.0, 1.0, 0.5, 0.1, 0.01, 0.02)
Size of negative set |{d0 }|2

For the phrasal scoring model, we copied the model settings from [11] asshown in Table 1. For word embeddings, we use the pre-trained GloVe1 which
was trained on data from ”Common Crawl”2 , an open repository of web crawl
data. For computation of ROUGE scores, we used publicly available ROUGE
1

https://nlp.stanford.edu/projects/glove/ Common Crawl (840B tokens, 2.2M
vocab, cased, 300d vectors)
2
http://commoncrawl.org/

implementation3 with Poster stemmer and stopword removal enabled4 . We usedthe default word and sentence tokenization from NLTK[6]5 .
We reported our system’s validation results (Table 2) with the following
metrics:– MRR: Mean reciprocal rank.
– MAP: Mean average precision.
– P-macro, R-macro, F-macro: Precision, Recall, F-measure whose values areaveraged by query. This is straightforward as we average the results of all
folds in the leave-one-out validation.
– P-micro, R-micro, F1-micro: Precision, Recall, F1 whose values are computedby putting the outputs of all test queries in the leave-one-out validation into
one for micro-based measurement.
The validation results (Table 2) show that performance is improved whenusing encoded summarization over lexical matching and even further improved
when combining both of them. The combination improve 0.185 F1-macro and0.182 F1-micro compared to lexical matching and 0.105 F1-macro and 0.091 F1micro compared to encoded summarization. While the improvement from lexicalmatching to encoded summarization suggests the importance of summarizationin supporting case retrieval, the further improvement of the combination hints
the existence of latent features not captured by lexical approach.

Table 2. Leave-one-out validation results on provided 285 training queries. Precision(P), recall (R) and f-measure (F1) are calculated with candidate selection strategy 1:
top k = 10 (the average number of supporting cases in training data).
P
R
F1
P
R
F1
macro macro macro micro micro micro

Model

MRR MAP

Lexical Matching

0.764 0.530 0.420

0.520

0.398

0.420 0.426 0.423

Encoded Summarization

0.874 0.659 0.510

0.584

0.478

0.510 0.517 0.514

Lexical Matching
0.966 0.849 0.601+Encoded Summarization

0.761

0.583

0.601 0.609 0.605

The submission has two folds corresponding to the two candidate selection
strategies. We selected k = 10 which is the average number of noticed cases
from the training data and r = 2.5 which results in the best F1-macro in thevalidation process (Fig. 1). The submission results are shown in Table 3.
3

https://github.com/andersjo/pyrouge
rouge parameters: -c 95 -2 -1 -U -r 1000 -n 2 -w 1.2 -a -d -m -s
5https://www.nltk.org/ version 3.3.0
4

0.7
0.53940.6

0.5471
0.48800.47780.5

F10.57610.5759

0.4
0.3
0.2

0.1
0

0.5

1

1.5

2

2.5

3

3.5

4

r
Fig. 1. Leave-one-out validation results of candidate selection strategy 2 on provided
285 training queries. We evaluated each r ∈ {1, 1.5, ..., 3.5} based on the competition’s
primary measure F1.

Table 3. Submission results on test data. We submitted the outputs of the combination
model “Lexical Matching + Encoded Summarization”. The submission has two folds
corresponding to the two candidate selection strategies. We selected k = 10 which isthe average number of noticed cases from the training data and r = 2.5 which resultsin the best F1-macro in the validation process (Fig. 1).
Submission

P-microR-microF1-microk = 10

0.675

0.634

0.655

r = 2.5

0.546

0.655

0.596

3

Statute Law Retrieval Task

3.1

Task Overview

A query in the Statue Law Retrieval task is usually represented as a statementthat a user want to check whether or not it is correct. The task will try findarticles in Japanese Civil Code that are relevant to the query to answer it. Thisis also a challenging task. Firstly, in many cases, articles describe the generallaw provision, but the query describes a concrete situation. Therefore, the lexicalmatching methods may fail to find relevant articles. Secondly, in some cases,
although the query are quite similar to the article by lexical matching method,they are not relevant. Table 4 shows an example in which the article A961
shared much more words than article A974, however, it is not relevant to thegiven query.

Table 4. An example that lexical matching methods fail to find relevant articles
QUERY: A minor who has attained 15 years of age may be a witness to a will
ARTICLELABEL* A961: Any person who has attained 15 years of age may NOT RELEVANT
make a will.
* A974: The following persons may not be a witness or RELEVANT
observer to a will ...

We found that in these cases, the effectuation parts of articles (the boldparts in the example) are more important than requisite parts in measuring therelevance between the query and articles. Our proposed system also focuses thisissue by using the effectuation parts as additional information to compute therelevance.

3.2

Statute Law Retrieval based on Requisite-Effectuation Analysis

We use the conventional Information Retrieval approach to solve this task withthe usage of some additional information which obtained from analyzing components of legal sentences. Requisite and effectuation parts are main components
in legal sentences and we hypothesize that these parts may affect to the performance of IR task. The system is based on [10] with several steps:
Processing : We conduct some basic processing techniques such as stemming,and removing stop-words.

Indexing : We using uni-gram, bi-gram, three-gram word indexing model with
the TF-IDF term weighting technique to represent articles as vectors. We use
two different scopes to index articles:– Indexing based on the whole content of each article.
– Indexing based on effectuation parts: We use only effectuation parts of eacharticle. Effectuation parts of articles are obtained by analyzing JPC-RRE
corpus [7].
Scoring models and Relevant Analysis: We compute cosine similarity scores between vector representations of the given query and each article as follows:– s1 (q, a) = cosine(q, a) : computes the similarly between the query q and thewhole content of the article a.
– s2 (q, a) = cosine(q, aE ) : computes the similarly between the query q andeffectuation parts of the article a.
– similarity(q, a) = s1 (q, a)+α×s2 (q, a): This is the combination of two above
scores. The value of α indicates how much the contribution of effectuation
parts. If α is 0, this is the same with original method which described in[10].
After similarity scores are computed, a number of top-k relevant articles are
selected as relevant articles.
3.3

Experiments and Results

Parameters tuning We use the training set for tuning parameters and model
selection. Firstly, we tune the indexing model and top-k, the left figure in Fig. 2
shows that the bi-gram word indexing model and top − k = 2 produce the best
performance (in F2 measure). Secondly, we tune the parameter α to find thathow much how much the information of effectuation parts will be add into tooriginal score to obtain the best performance. The right figure in Fig. 2 showsthat the best combination of two similarity score is obtained when α = 0.4.
Results Table 5 shows the evaluation result on the official test set which includes
69 queries. Although the effectuation parts show positive effects in the training
set, it reduces the performance on the test set. However, it improves the MAP
scores.

Top k and indexing model selection

Tune ⍺ with 2-gram indexing modeland top-k retrieval articles

0.56

0.536

0.54

0.534

0.52

0.532
0.53

0.48

0.528

0.46

0.526

F2

F2

0.5

0.524

0.44

0.522

0.42

0.52

0.4
1

2

3

4

5

Top k
1-gram2-gram

0.518
0.516

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1⍺

3-gramFig. 2. Hyper parameter tuning including (1) tuning top-k and indexing model (leftfigure) and (2) tuning α for scoring model. The best parameter set are {top − k =2, indexingmodel : 2 − gram, α = 0.4}

Table 5. Performance on the test set. Different values of α indicate the contributionsof effectuation parts in computing similarity scores.
Submission

Precision

RecallF2

α = 0.0
α = 0.4

0.413
0.40580.64040.62920.57690.5668

Submission

R@2

R@5

R@10

R@30

R@100 MAP-L

α = 0.0
α = 0.40.64040.62920.764
0.75280.82020.809

0.92130.91010.95510.97750.73980.7296

4

Conclusion

We have presented our methods for legal case retrieval and statute law retrieval
tasks in Competition on Legal Information Extraction/Entailment, 2018.
For the legal case retrieval task, we developed an approach of combining
lexical features and latent features embedding summary properties which wecall encoded summarization. The validation results show that performance isimproved when using encoded summarization over lexical matching and evenfurther improved when combining both of them. While the improvement fromlexical matching to encoded summarization suggests the importance of summarization in supporting case retrieval, the further improvement of the combination
hints the existence of latent features not captured by lexical approach.
For the law statute retrieval task, we incorporated the structure of legal
sentences in computing similarity between queries and articles. However it only
show the improvement on the training set. In future, we will investigate different
types of additional information such as subjects, verbs and objects of legal sentences because each of type may possess different contributions for the relevant
analysis in this task.

Acknowledgements
This work was supported by JST CREST Grant Number JPMJCR1513, Japan.

References
1. Duchi, J., Hazan, E., Singer, Y.: Adaptive subgradient methods for online learningand stochastic optimization. Journal of Machine Learning Research 12(Jul), 2121–
2159 (2011)
2. Johnson, R., Zhang, T.: Effective use of word order for text categorization withconvolutional neural networks. In: Proceedings of the 2015 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 103–112. Association for Computational Linguistics, Denver, Colorado (May–June 2015), http://www.aclweb.org/anthology/N15-1011
3. Kalchbrenner, N., Grefenstette, E., Blunsom, P.: A convolutional neural network
for modelling sentences. In: Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 655–665.
Association for Computational Linguistics, Baltimore, Maryland (June 2014),
http://www.aclweb.org/anthology/P14-1062
4. Kim, Y.: Convolutional neural networks for sentence classification. In: Proceedingsof the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP). pp. 1746–1751. Association for Computational Linguistics, Doha, Qatar(October 2014), http://www.aclweb.org/anthology/D14-1181
5. Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out (2004)
6. Loper, E., Bird, S.: Nltk: The natural language toolkit. In: Proceedings of the ACL02 Workshop on Effective Tools and Methodologies for Teaching Natural Language

Processing and Computational Linguistics - Volume 1. pp. 63–70. ETMTNLP ’02,
Association for Computational Linguistics, Stroudsburg, PA, USA (2002), https://doi.org/10.3115/1118108.1118117
7. Nguyen, T.S., Nguyen, L.M., Tojo, S., Satoh, K., Shimazu, A.: Recurrent neuralnetwork-based models for recognizing requisite and effectuation parts in legal texts.
Artificial Intelligence and Law 26(2), 169–199 (Jun 2018), https://doi.org/10.
1007/s10506-018-9225-1
8. Pennington, J., Socher, R., Manning, C.D.: Glove: Global vectors for word representation. In: Empirical Methods in Natural Language Processing (EMNLP). pp.
1532–1543 (2014), http://www.aclweb.org/anthology/D14-1162
9. Severyn, A., Moschitti, A.: Learning to rank short text pairs with convolutional
deep neural networks. In: Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval. pp. 373–382. ACM(2015)
10. Son, N.T., Anh, P.V., Minh, N.L.: Recognizing entailments in legal texts using
sentence encoding-based and decomposable attention models. In: Satoh, K., Kim,
M.Y., Kano, Y., Goebel, R., Oliveira, T. (eds.) COLIEE 2017. 4th Competition on
Legal Information Extraction and Entailment. EPiC Series in Computing, vol. 47,
pp. 31–42. EasyChair (2017), https://easychair.org/publications/paper/wHFc
11. Tran, V.D., Nguyen, M.L., Satoh, K.: Automatic catchphrase extraction from legal
case documents via scoring using deep neural networks. In: Workshop on MIning and REasoning with Legal texts (2018), https://www.researchgate.net/
publication/327645790_Automatic_Catchphrase_Extraction_from_Legal_
Case_Documents_via_Scoring_using_Deep_Neural_Networks