HUKB at COLIEE2018 Information Retrieval
Task
Masaharu Yoshioka1,2 and Zihao Song1
1

Graduate School of Information Science and Technology, Hokkaido University, N14
W9, Kita-ku, Sapporo-shi, Hokkaido, Japan
yoshioka@ist.hokudai.ac.jp
2
Global Station of Big data and cybersecurity, Hokkaido University, N14 W9,
Kita-ku, Sapporo-shi, Hokkaido, Japan

Abstract. In COLIEE-2018, there are four tasks as a combination of
document domains (case law and statute law) and task types (information retrieval (IR) and entailment) and we (HUKB) participated in the
IR tasks of two document domains (Tasks 1 and 3). In this paper, we
introduce our system for the submission of final results and discuss thecharacteristics of the system.
Keywords: Legal Information Retrieval · Query Analysis · Query Construction.

1

Introduction

In Competition on Legal Information Extraction/Entailment (COLIEE)-2018,
there are four tasks as a combination of document domains (case law and statutelaw) and task types (information retrieval and entailment). In this COLIEE2018 3 , we (HUKB) participated in the information retrieval (IR) task of two
document domains (Tasks 1 and 3).
In this paper, we introduce our system for the submission of final results.
For Task 1, we mainly discuss the method of handling structural information ofcase law for query and database construction. For Task 3, we mainly discuss themethod of identifying diﬃcult questions to use diﬀerent approaches for selecting
relevant articles. We also discuss the characteristics of the system by using theevaluation results of the training data and test data.

2

Case Law Retrieval Task

2.1

System architecture

Because the case law retrieval task (Task 1) was newly introduced, there was no
general approach of constructing an IR system for the task.
In this framework, we built an IR system for the task by using the following
two steps to retrieve referred cases from the database.
3

https://sites.ualberta.ca/￣ miyoung2/COLIEE2018/

2

M. Yoshioka et al.

1. Ranked retrieval
We use an IR system to generate ranked retrieval results.
2. Selection of referred cases
By using the ranked retrieval results, we select the final answer from them.
For the ranked retrieval results, we conducted experiments to address thefollowing criteria.
1. How to handle structural information of case law.
Case law texts consist of multiple parts (summary, fact, citation, and others).
We discuss which part of the text is useful to make the index and query.
2. Two database construction methods to use candidate documents for IR.
In the training data, there were 200 similar candidate case documents foreach query. However, to obtain better term distribution information among
all case laws, it is preferable to have a larger text database. We constructed
two databases; one was constructed of candidate documents only, and theother of all candidate documents. For the latter database, the final results
were selected from candidate documents.
For the selection of referred cases, we propose the following two simple methods.
– Fixed number selection
The system returns a fixed number of cases for all queries. The number ofreturned cases is decided based on the preliminary experiments using testdata.
– Variable number selection
The system selects related cases by checking the similarity with unrelatedcases. In this experiment, we assumed that cases for all other queries wereunrelated cases.
2.2

Case law text analysis

The text structure of case law is not well formalized, but most of the text has asimilar style. We extracted structural information based on pattern matching ofa section, such as lines in the text.
First, we classified the text into three parts, namely header, facts, and footer.
The beginnings of the facts are identified by a fact representative line that starts
with number 1 in brackets (i.e., [1]). The end of the facts part is identified by
the lines Editor: or [End of document].
From each part, we extracted the following information.
– Header parts
Header parts contain a variety of information, such as reference information, name of the court, summary, counsel, solicitors of record, etc. We
extracted the summary information by checking the line that starts with
Summary: and extracted text until the line with other heading keywords

HUKB at COLIEE2018 Information Retrieval Task

3(e.g., Administrative Law, Statues Noticed, and Counsel). If the systemfails to extract summary parts from the header, the system uses all header
text of a case as a summary.
– Fact parts:
Several documents had facts part in two languages (English and French). In
most of the cases, the French parts were followed by English and the same
numbers were used. So, we excluded text parts after the appearance of thesecond [1].
– Footer parts:
We did not use footer parts for the experiment.
By using this simple text extraction framework, a summary and a fact partwere extracted from each candidate document.
To discuss which parts of the texts are useful to identify related cases, weused three diﬀerent text parts to construct a document database, as follows.
– P artsum : Summary only
– P artf act : Fact only
– P artall : All (summary and fact).
In addition, because query texts were given as summary.txt and fact.txt,
we could construct three types of queries as follows.
– Qsum : summary.txt– Qf act : fact.txt– Qall : summary.txt and fact.txt.
2.3

Database construction with two diﬀerent document sets

The original IR task was designed to select referred case documents from 200
candidate documents. Although there is no information on how to select 200 such
documents from all the case documents, the statistical information of the terms(e.g., document frequency, term collocation information) for these candidates
might not be the same as that for all case laws. As a result, the statistical information extracted from candidate documents only may have problems (e.g., whencandidate documents are selected from the existence of important keyword(s) in
the query, the inverted document frequency (IDF), such as importance measuresof those terms, are discounted compared with the case with all case documents).
Therefore, we constructed two databases for IR.
– DBeach : The database for the IR system was constructed by using candidate
documents only (we constructed a database for each query).
– DBall : The database for the IR system was constructed by using all documents. For training analysis, we used documents for training data only,
but for the final submission, we used all documents (training and test) for
database construction.
For the latter case, because IR results that use DBall may return documentsthat are not included among the candidates, we applied postprocessing to selectcandidate cases for the query from the result.

4

M. Yoshioka et al.

2.4

Experimental analysis using training data

First, we discuss how to generate ranked retrieval results with two discussion
criteria (usage of structural information and database construction with diﬀerent
document sets) by using a preliminary experiment with 18 diﬀerent parametersettings (3 Parts × 3 Queries × 2 DBs).
We used Indri for the basic IR system with out-of-the-box settings. Indri isa state-of-the-art IR system based on language modeling [3] 4 and the retrieval
performance of Indri is also good for COLIEE Statute Law retrieval with thesame settings [4].
Table 1 shows the mean average precision (MAP) of the retrieved resultswith diﬀerent parameter settings (3 Parts × 3 Queries × 2 DBs).
Table 1. MAP of diﬀerent parameter settings
Text type and Query type DBeach DBall
P artall +Qall
0.3616 0.3852
P artall +Qsum
0.4044 0.4284
P artall +Qf act
0.3498 0.3732
P artsum +Qall
0.3488 0.4020
P artsum +Qsum
0.3591 0.3981
P artsum +Qf act
0.3298 0.3827
P artf act +Qall
0.3567 0.3801
P artf act +Qsum
0.3972 0.4240
P artf act +Qf act
0.3437 0.3684

From these results, we confirmed that P artall +Qsum +DBall were the best
results from among these settings. In addition, P artall +Qsum +DBall significantly outperformed other settings statistically (Wilcoxon signed rank test p =0.01), except for P artf act +Qsum +DBall . The following summarizes this experiment.
– It is better to use a larger document set (DBall ) to extract statistical information of the terms.
– It is better to use summary text (Qsum ) for query construction. Summary
text may cover all important keywords and fact text may contain unnecessary
keywords that are not appropriate for the query.
– It is better to use all text (P artall ) for text database. Using summary text
only may miss keywords that exist in the query. Although the performancewith using fact only (P artf act ) is close to that using all text (P artall ), it
may be reasonable to use all text.
After this experiment, we conducted all experiments using the P artall +Qsum +DBall
setting.
For the selection of referred cases, we conducted the following experiment.
4

https://www.lemurproject.org/indri/HUKB at COLIEE2018 Information Retrieval Task

5

– Fixed number selection
Based on the preliminary experiment, we generated a list with a fixed number of returned cases. From comparison, we selected eight for fixed number
selection.
– Variable number selection
We assume that the related cases are highly ranked with other cases; therefore, when the number of the related cases is small, the number of cases thatbelong to other candidate cases increases. Therefore, the number of cases forthe question that exists in the highly ranked retrieved results may decrease.
Therefore, when we control the number of documents to check whether the
document belongs to the candidates, the number of cases selected from thelist is controlled. In this experiment, we introduced two parameters; one is
the minimum number of cases Candmin that should be included for the retrieval results, and the other is the maximum number of cases Checkmax thatare checked to verify whether documents belong to the candidates. When the
checked retrieved results reach Checkmax and the number of cases to selectis below Candmin , the system checks the list to find Candmin cases using
the ranked list. We conducted experiments with diﬀerent parameter settings
using the test data and decided that Candmin = 3 and Checkmax = 40 inthis experiment.

2.5

Submitted results

We submitted two runs that use a variable number selection setting (HUKB1)
and a fixed number selection setting (HUKB2).

Table 2. Evaluation results of submitted runs and best run (Task 1)
runnumber of returned documents precision recall
F
HUKB1
390
0.4974 0.3084 0.3808
HUKB2
472
0.4047 0.3037 0.3469
JNLP-k=100.6763 0.6343 0.6546

From this result in Table 2, since HUKB1 has a higher evaluated value forboth precision and recall, we confirm that it is better to have a mechanism that
controls the number of returned cases for each query. However, the performanceof the system is not as good as the best performance system (JNLP-k=10).
However, there is no explanation about how to construct a candidate caseset. If it is diﬃcult to construct such candidates automatically, the result of
HUKB1 might not be reproducible. In such a case, it is necessary to consider
other methods for controlling the number of returned cases.

6

M. Yoshioka et al.

3

Statute Law Retrieval Task

3.1

System architecture

We already participated in the statute law retrieval task of COLIEE [2, 5] andour system for the statute law IR task is an extension of the previous system.
The following is the basic system architecture of our previous system [5].
1. The system uses the output of IR results with diﬀerent settings as a featurefor final rank calculation using SVM rank [1].
2. Train the ranking model with diﬀerent settings and merge the results basedon their outputs.
Diﬀerences from the previous system are as follows.
1. Manual construction of database of conditions and decisions for civil law
articles. We manually checked all civil law articles to extract the condition
parts (for judicial decisions) of the articles and their judicial decisions toconstruct an article database of the condition and decision parts. The system
extracts the condition and judicial decision parts from the question and uses
them as queries to calculate similarity between questions and articles.
2. Using Indri [3] instead of BM25.
As we confirmed, Indri achieved good performance even for out-of-the-box
settings [4]; we use Indri for a basic IR engine to calculate the similarity
score among queries and articles. To make an index of Indri, we used the
MeCab morphological analyzer to split the sentence into a word sequence. We
used the same procedures for splitting questions into query word sequences.
Details of these procedures are explained in [4].
3. Introducing a new system to identify topic diﬃculty.
Based on the analysis of the performance of all COLIEE participants [4],there are two types of queries: easy queries for which almost all participants
can select at least one related candidate article for the query, and diﬃcult
queries for which almost all participants cannot retrieve related articles. We
introduce a machine learning-based system to identify such topic diﬃculty.
Details of these diﬀerences are discussed in the following subsections.
3.2

Extraction of condition and decision parts

As the articles use multiple sentences and enumerations to describe the conditions of the judicial decisions, it is not very easy to classify this information using
a simple syntactic analysis. Therefore, we manually checked all the articles ofJapanese civil law to select the condition and judicial decision parts.
Compared with the articles, it is not very diﬃcult to extract the conditionand judicial decision parts from the questions using the dependency parser KNP
5
and clue keywords. 1 shows an output of KNP for the question in the training5

http://nlp.ist.i.kyoto-u.ac.jp/EN/index.php?KNPHUKB at COLIEE2018 Information Retrieval Task

7

data (H18-26-1). ”ＡＢが甲建物を持分各２分の１の割合で共有していた場合，Ａが死亡して相続人も特別縁故者もいないときは，甲建物の所有権はＢに帰属する。”(”In cases where person A and person B co-own building X at the ratio of 1:1, if
person A dies and had no heirs or persons with special connection, the ownershipof building X belongs to person B.”) that belongs to the former case. From thisdependency tree, we select the condition part branches by using keywords suchas ”場合 (in case)”, and ”とき (if)”. In this case, we select branches that end in”場合 (in case)”, and ”とき (if)” (rounded rectangular with thick dotted line)
and select ”ＡＢが甲建物を持分各２分の１の割合で共有していた場合，Ａが死亡して相続人も特別縁故者もいないときは，(In cases where person A and person B
co-own building X at a ratio of 1:1, if person A dies and had no heirs or persons
with special connection)” (rectangular with thin dotted line) as a condition partand ”甲建物の所有権はＢに帰属する。 (ownership of building X belongs to person
B.)” as the decision part.

Fig. 1. Result of KNP parsing

3.3

Ranked retrieval system using SVM rank

This year’s system uses Indri for generating ranked retrieval results as featuresof SVM rank.
We used the following features for SVM rank.

8

M. Yoshioka et al.

1. Indri scores with various pairs of queries and text parts
We constructed three types of queries (condition part, decision part, all text).
We also constructed three document databases that contain a part of a document (condition, decision, title, and contents) and one document databasewith all text (title and contents). We calculated Indri-based similarity scoresfor combinations of all queries and all document databases except the condition query with the decision database and the decision query with thecondition database) (3 × 5 − 2 = 13 similarity scores are calculated).
2. The longest common subsequence with various pairs of queries and text parts
For the same 13 variations, scores based on the longest common subsequence
proposed in the previous system [5] were calculated.
As a result, we have 26 features for calculating similarities between questions
and articles. All feature values were normalized to the 0–1 scale using maximum
and minimum values using linear transformation (maximum values correspondto 1 and minimum values correspond to 0).
To generate diﬀerent variations of models for checking the diﬃculty of the
problem, we constructed 11 variations of models for SVM rank by excluding one
year data (H19?H28, H18+H20-H28, ..., and H18-H27) for training data. In thisyear, we use the SVM rank software 6 based on scikit-learn. 7
In addition, we also used each model to check the diﬃculty of the question. We
conducted a retrieval for cross validation setting (e.g., evaluated H18 questions
using a model that excludes H18 data for training) and checked the rank of all the
relevant articles. When the ranks were larger than 50, we classified these results
as outliers for training the model to retrieve easier questions. The following
are diﬃcult questions that were excluded as outliers H18-23-I, H18-23-U, H1827-U, H18-29-4, H19-15-I, H21-24-E, H22-15-A, H22-27-2,H24-19-1, H24-22-3,
H24-27-A, H24-30-3, H25-15-4, H25-21-O, H26-14-I, H26-16-E, H26-27-C, H2627-I, H26-32-G, and H28-14-1. In most cases, these questions require knowledge
about mapping between keywords. For example, in the case of H18-23-I ”Ａが
宝石をＢに売り，その代金をＢがＣに支払うとの契約を締結し，Ｃが受益の意思表旨をした場合，Ａが宝石をＢに引き渡したが，Ｂが代金をＣに支払わないとき
は，ＣはＢに対して代金を自己に支払うよう請求することができるが，ＡもＢに対して代金をＣに支払うよう請求することができる。”(”In cases where person
A sold a jewel to person B, person B entered into a contract where person B
was assigned to pay the purchase money to person C, and person C expressed
his/her intention to enjoy the benefit of the contract, if person A delivered thejewel to person B, and person B has not paid the purchase money to person C
yet, while person C can demand that person B should pay the purchase money,
person A can also demand that person B should pay the purchase money toperson C.”), it is necessary to understand that ”支払う (pay)” and ”意思表旨(express intention)” of the question corresponds to ”給付 (tender)”, and ”約す
る (promise) in the articles to retrieve article 537. We assume that these queries
67https://gist.github.com/agramfort/2071994
http://scikit-learn.org/

HUKB at COLIEE2018 Information Retrieval Task

9

are diﬃcult questions discussed in [4] and decided to treat them as outliers fortraining the SVM rank model for retrieving articles for easy questions.
After identification of such outliers, we regenerated the SVM rank model by
excluding such questions from the training data.
By using the output of 11 models, we obtained 11 diﬀerent ranking resultsfor each question. Then, the system calculated the average rank of each article
and sorted the articles using this average rank in ascending order to generate
final merged retrieval results.
3.4

Identification of diﬃcult questions

The identification of diﬃcult questions is a task to estimate the outliers discussed
in the previous section without using answer information.
Since topics become diﬃcult when there is term mismatch among questions
and articles, we assume that the similarity measure for first-ranked documentsof the retrieved results may be comparatively worse than for easy ones.
Therefore, we used the document similarity score information for the firstranked document (i.e., Indri scores and longest common subsequence-based scoreswith various pairs of queries and text parts are used for the features to identify
the topic diﬃculty and linear SVM of scikit-learn is used to classify the topic
diﬃculty).
For the diﬃcult questions, to include the relevant articles in the list, we
decided to include 100 articles to keep a better recall.
Another perspective of diﬃculty can be checked by the consistency among
the retrieval results of the 11 models. There were several cases in which the 11
models selected diﬀered in their top-ranked articles. In such a case, it may bebetter to include all top-ranked articles to increase the recall of the system.
3.5

Submission results

We submitted two diﬀerent versions for Task 3. One is a method that considers
the diﬃculties of questions (HUKB1) and the other selects top-ranked documents
only (HUKB2).
As the system cannot identify the diﬃcult questions by using SVM, theresults of HUKB1 and HUKB2 are almost the same except for the inclusion of
extra articles because of the inconsistency among the 11 models (five questions
have two articles for the retrieved results).
Table 3 shows the evaluation results of the submitted runs. As HUKB1 andHUKB2 use the same merged rank results, the MAPs of these two runs are thesame. As the added secondary ranked document using inconsistency among the
11 models did not include any related articles, the performance of HUKB1 wasworse than HUKB2. ORG1 used Indri, which used all the text of questions as
query and all parts of articles for constructing the text database. The resultof ORG1 was almost identical to HUKB2, except that HUKB2 found relevant
articles for H29-3-O and H29-6-E. For these two cases, ORGJ1 found the relevant
document as second-ranked, but HUKB2 found it as first-ranked.

10

M. Yoshioka et al.

Table 3. Evaluation results of submitted runs (Task 3) and corresponding organizers’
run
run return retrieved precision recall F2 MAP
HUKB1 7453
0.7162 0.5955 0.6163 0.7479
HUKB2 6953
0.7681 0.5955 0.6235 0.7479
ORGJ1 6951
0.7391 0.5730 0.6000 0.7398

From these results, our system improved slightly compared with simple Indri,
but the diﬀerence was rather marginal.
3.6

Discussion

Despite the eﬀort to improve the system performance using several techniques,
the final performance of the system is currently not very impressive.
There are several issues to be solved before discussing the appropriateness of
this approach.
1. Revision of the system to extract conditions and decisions
Based on the analysis of the final results of extracting conditions and decisions of the test data, there were several questions where the system failed to
extract the appropriate parts. The quality of extraction needs to be improved
and the results checked after the revision.
2. Analysis of diﬃcult questions
Evaluation of the appropriateness of the feature set used for diﬃcult question
identification is needed. Possible candidates for the additional features areones calculated by the score diﬀerence between the first-ranked documentswith other documents (e.g., second-ranked and tenth-ranked). If the firstranked documents are closely similar to the question (e.g., longer longest
common sequence), the scores of the first-ranked documents are comparatively higher than that of the second- or tenth-ranked documents. In contrast, if there is a smaller overlap between the first-ranked documents andthe question, the diﬀerence becomes smaller than in the former case.
However, if there are several bugs in the system that generate the final submission results, the system needs to be revised to check the performance basedon the new results.

4

Summary

In this paper, we introduced our system for participating in Tasks 1 and 3 of
COLIEE-2018. For the Task 1 system, we found that it was better to use larger
case data to estimate term statistical information and the summary part wasmore important than the fact part to construct the query. We also confirmed thatthe number of cases needs to be controlled. For Task 3, there was no significant
diﬀerence compared with the basic IR system. The appropriateness of the systemneeds to be checked to decide whether the proposed approach is promising.

HUKB at COLIEE2018 Information Retrieval Task

11

Acknowledgement
This work was partially supported by JSPS KAKENHI Grant Number 16H01756
and 18H0333808.

References
1. Joachims, T.: Training linear svms in linear time. In: Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 217–226. KDD ’06, ACM,
New York, NY, USA (2006). https://doi.org/10.1145/1150402.1150429,
http://doi.acm.org/10.1145/1150402.1150429
2. Onodera, D., Yoshioka, M.: Civil code article information retrieval system based on
legal terminology and civil code article structure. In: The Proceedings of the 10th
International Workshop on Juris-Informatics (JURISIN2016) (2016), paper 19
3. Strohman, T., Metzler, D., Turtle, H., Croft, W.B.: Indri: A language model-based
search engine for complex queries. In: Proceedings of the International Conferenceon Intelligent Analysis. pp. 2–6 (2005)
4. Yoshioka, M.: Analysis of coliee information retrieval task data. In: Arai, S., Kojima,K., Mineshima, K., Bekki, D., Satoh, K., Ohta, Y. (eds.) New Frontiers in Artificial
Intelligence. pp. 5–19. Springer International Publishing, Cham (2018)
5. Yoshioka, M., Onodera, D.: A civil code article information retrieval system basedon phrase alignment with article structure analysis and ensemble approach. In:
Satoh, K., Kim, M.Y., Kano, Y., Goebel, R., Oliveira, T. (eds.) COLIEE 2017.
4th Competition on Legal Information Extraction and Entailment. EPiC Series
in Computing, vol. 47, pp. 9–22. EasyChair (2017). https://doi.org/10.29007/5zzj,
https://easychair.org/publications/paper/PRH