Question Answering System for Legal Bar Examination
using Predicate Argument Structure
Reina Hoshino1, Ryosuke Taniguchi1, Naoki Kiyota1 , and Yoshinobu Kano1
1 Informatics Course, Graduate School of Integrated Science and Technology,

Shizuoka University, 3-5-1 Johoku, Naka-ku, Hamamatsu City, Japanrhoshino@kanolab.net, rtaniguchi@kanolab.net
nkiyota@kanolab.net, kano@inf.shizuoka.ac.jp

Abstract. We developed a question answering system for legal bar exam,
which can explain the way system solves based on underlying logical structures. We focus on the set of subject and object with their predicate, i.e. thepredicate argument structure, in order to represent structures of legal documents. We implemented a couple of modules using different searching methods. Our system outputs results using these modules by learning each module’s
confidence value with SVM. We manually analyzed the difficulty level of the
problems whether external knowledge is required or not. We created a structured synonym dictionary specialized to the legal domain, where predicates are
categorized with their objects. This synonym dictionary could absorb superficial differences of predicates to solve the problems which do not require external knowledge. We confirmed that the system can solve more than 70 percent
of simple problems. Our system achieved the second best score in Task 4 of the
COLIEE 2018 shared task.
Keywords: COLIEE, Question Answering, Legal Bar Exam, Legal Information
Extraction, Predicate Argument Structure Analysis.

1

Introduction

Automatic question answering for legal documents is gathering attention recently.
This paper describes our challenge to Task 4 of the COLIEE 2018 legal bar exam,
which asks participants to answer true or not based on the Civil Law Articles, given
text drawn from the Japanese legal bar exam.
The COLIEE shared task series is held in association with the JURISIN (Jurisinformatics) workshop. The first one was the COLIEE 2014 shared task[1]. Following
this, COLIEE 2015 shared task[2], COLIEE2016 shared task[3], and COLIEE
2017[4].shared task (this time in conjunction with ICAIL) were held. The COLIEE
shared task consists of four tasks. We challenged Task four. Task one and Task two
used the legal cases. Task one of this legal question answering task involves reading alegal bar exam question, and extracting a subset of Japanese Civil Code Articles. Task
four requires both of the legal information retrieval system and textual entailment
system. Given a set of legal yes/no questions, a participant’s system will retrieve rele-vant Civil Law articles. Then answer yes/no entailment relationship between input
yes/no question and the retrieved articles. The corpus of legal questions is drawn fromJapanese Legal Bar exams, and the relevant Japanese Civil Law articles were also
provided. While there was an English translation version of the dataset provided, weonly used the original Japanese version.
Regarding question answering in general, most successful systems employ rather
traditional techniques of question answering which have decades of history [5]–[8].
Recently, automatic answering system used machine learning by big data. The typicalexample is IBM Watson System [9]. The core Watson system employed a couple ofopen source libraries, including the traditionally well-designed DeepQA system [10]as its skeleton of question answering processing.
However, there are only hundreds of problem sentences used in COLIEE. We thinkthat the amount of data is not enough to perform an end-to-end supervised machinelearning. There are an enormous number of legal documents, many methods havebeen proposed to acquire knowledge from case examples [11]. However, it is difficult
to use case examples directly in the legal bar exam, because we need commonsense
knowledge to follow the thinking of a legal expert, not just the explicit knowledge
base available. In addition, a question answering system for legal documents is required to show reasons of its decision in a human interpretable way. Our aim is tocreate a legal question answering system where we can trace evidences of it decision.
Reasoning is essential in understanding a legislative system. Civil law articles consist of a legal effect part and cases to state its effect. A system needs to find themfrom problem sentences. However, there are a couple of difficult issues to find such
relevant parts.
Firstly, the legal effect is not necessarily written in the article. We conducted anexperiment asking Japanese native speaker to solve the legal bar exam while showing
the related articles. These native speakers have never learned legal issues. Their accuracy was about 80 percent, which shows that people have little knowledge of law
cannot solve all problems even the related articles are given. This means that external
knowledge is important to solve the problems.
Secondly, there are ambiguities in legal articles. Words used in the law articles aredifferent from the words in problem sentence in most cases, even when they express
the same situation. For example, “成立する (effect)” and “適用する (apply)” aresometimes interchangeable. We created a synonym dictionary for predicates to hadle
this issue. We made our synonym dictionary extracting from the past legal bar exam
problems of six years (2009-2014). A predicate entry in our dictionary is structured tohave a condition, what sort of object could be taken.
Thirdly, diversity of the vocabulary is a difficult issue. In the civil law, both legal
technical terms and terms used in daily life are required, because civil law handles
problems in daily life. We registered these words in our morphological analyzer’s
dictionary..
The legal bar exam includes a variety of complex linguistic issues, such as synonym, commonsense knowledge, syntax, and semantics. to be resolved to understandlegal documents. In order to make our system behavior interpretable, we focus on

3

simpler problems in this paper. We manually examined difficulty levels for tens ofpast years’ problems.
Section 2 describes the proposed method of our question answering system. Section 3 shows results of COLIEE2018 and past years, including analyses of the difficulty levels. Section 4 concludes this paper, discussing future works.

2

System Architecture

2.1

System Design

Our question answering system consists of two parts: a related articles search part,and a question answering part. These parts use predicate argument structure analysis,comparing a pair of sentences based on case roles of the arguments. The article searchpart searches for the related articles by searching sentences of the same structure. The
question answering part compares whether each sentence represents the same event.
In addition, we prepare a couple of different modules with different judgment criteria,
and make modules for each criterion. Our final answer is obtained by using SVM,
which selects its output from the answers of the modules and their confidence values.
These confidence values are calculated from the number of articles that our predicate
argument matches with the problem sentences. The datasets we used were the civil
law articles, the problems of the past legal bar exam.
2.2

Dictionary

We created a legal term dictionary extracted from “有斐閣法律用語辞典第4版”(Yuhikaku legal term dictionary fourth edition).
Table 1. Number of words used in civil bar exam

H18(2006)
H19(2007)
H20(2008)
H21(2009)
H22(2010)
H23(2011)
H24(2012)
H25(2013)
H26(2014)
H27(2015)
H28(2016)

total
351
317
322
402
316
312
484
425
563
490
441

new
175
144
171
87
73
127
50
144
84
67

We examined vocabulary using past eleven years of problems, We found that morethan 300 kinds of words are used in problems per one year. More than 50 new words
appeared per year when registering words from older year to newer year (Table 1).
We registered these words in our morphological analyzer’s dictionary..
In addition to the dictionary above, we created an additional dictionary from morphological analysis results of the problems of 7 years. This dictionary makes ouranalysis of predicate argument structures correct. Predicate argument structure analysis often failed because morphological analysis was not correctly performed. This
failure is due to unknown words which are not listed in our Japanese dictionary nor
our legal term dictionary. We added 200 new words manually.
ｓWe created our additional dictionary, extracting from source codes of PROLEG(Satoh, Asai, & Hurukawa, 2011). PROLEG is a logic programming language for thelegal domain based on Prolog. They systemize internal structure of the law from thecivil law articles and actual cases. The PROLEG system returns a case is legally valid
or not, by giving arguments such as a defendant and an object. Function names andvariable names of PROLEG are named by hand. These names are necessary whenreading and understanding the legal documents deeply. Therefore, we could obtaincomposite words not listed in the normal legal term dictionary. We extracted function
names and variable names that do not use alphabets nor numbers, then added them toour additional dictionary.
2.3

Predicate Argument Structure Analysis

Fig. 1. An example of predicate Argument Structure Analysis

Fig. 1 shows an example of predicate argument structure analysis. We defined ourown clause unit (“節”) in order to recognize condition clauses and main clauses precisely, which are included in a single sentence. A clause should include a single predicate as a core element of that clause. We apply a dependency parser that makes
chunks (“文節”) of a couple of morphemes. Starting form a chunk that includes a5predicate, we aggregate neighboring chunks when a neighboring chunk does not include any predicate, until a clause unit is formed.
A predicate is not always suitable to be a core predicate of a clause. For example,“holding” in “condition holding a court” could be regarded as a predicate. However,this is not suitable as a core single predicate in a clause because we need to compare
larger predicate-argument structures rather than such a noun phrase.
We define two types of special clauses: a proposition clause and a condition clause.
A proposition clauses includes an end of the sentence. In the Japanese language, aclause which includes an end of sentence often represents a proposition. We regard aclause as a condition clause when that clause includes specific patterns,
e.g. ”when…”, ”in case of …”, etc.
When searching related articles, we use a set of a predicate, a subject and an object
as a predicate argument structure. For each sentence, we compare proposition clausesof the problem sentences and the civil law articles using these sets. The same applies
for condition clauses. We use the base form of predicates for their comparison. For
example, “認める (admit)” and “認めない (do not admit)” have the same meaning,
sharing the same base form “認める (admit)”.
Normally, a sentence consists of a proposition clause and any number of condition
clauses. Because an article consists of one or more sentences, we compare sentences
one by one when comparing the article sentences with the problem sentence. However, when a problem consists of a couple of sentences, we regard the whole sentences
as a single sentence. This is because the last sentence tends to be most important,while other sentences tend to represent conditions of the last sentence. For this reason,
we regard clauses as condition clauses except for the proposition clause of the lastsentence.
A difficulty in predicate argument structure analysis is an itemized article. An
itemized article first describes the effect, then cases are listed to show the effect. In
such a case, we cannot understand only with the first sentence. Sometimes there areno subjects or predicates in the part of case list. There are a lot of articles that havesuch a structure.
As a countermeasure, we connect the case list part with the first sentence. In thiscase, the first sentence always contains a phrase “次に掲げる (the following things)”.
We delete this specific phrase and then insert the part of case list. We repeat this process for the number of items, making multiple sentences. However, the connected
sentence could be syntactically invalid, causing errors in predicate argument structure
analysis.

2.4

Synonym Dictionary for PredicatesFig. 2. An example of synonym dictionary

We manually created a synonym dictionary for predicates. For example, “成立する(effect)” and “適用する (apply)” could have the same meaning. We used the problems of 2009 to 2006 to make our dictionary. We compared related articles and corresponding problem sentences to manually determine our synonym dictionary entries.
As a result, we selected a combination of 33 verbs.
Our dictionary is structured with object conditions. For example, “果実を取得する (obtain fruits)” and “収益する (receive the profits)” have the same meaning when
they have the same object.
We examined difficulty levels of the problems in order to verify how many problems can be solved by a synonym dictionary. The difficulty level is classified into
four levels. The first level is “very easy”, corresponds to the problems which text is
almost same with the article. The second level is “easy”, its problem can be solved by
any Japanese native speakers without external knowledge. This “easy” problem requires synonym and/or zero pronouns resolutions. The third level is “difficult”, its
problem requires general external knowledge but not legal expert knowledge. For
example, a father’s father is a grandfather, which is a common sense of external
knowledge. In addition, related articles in this level could span multiple sentences,
sometimes become complicated structures. The fourth level is “very difficult”, its
problem cannot be solved without external knowledge of the legal experts. In thislevel, its answer is not written in the related articles neither explicitly nor implicitly.
The “Target” column of Table 2 shows the difficulty level statistics of the problems in 2014 and 2016. The “easy” problems share around 30 percent. We aim tosolve these “easy” problems by our synonym dictionary.

7
Table 2. Difficulty level statistics of the past COLIEE problems

2014
2016

2.5

very easy
12
21

easy
10
32

difficult
530

very difficult
510Person Estimation

Some legal bar problems ask conceptual things, or ask how rights are oved using concrete stories. For example, a typical problem of concrete parable story describes persons called “A”, “B”, and “C”. In order to answer such concrete parable stories, we
implemented a person estimation feature. In our person estimation feature, our systemsearches for words that are actors in the sentence, i.e. words with “人(human)” or “者(person)”, e.g. “代理人(agent)”. Predicate argument structure analysis sometimes
fails to find a subject. When our system cannot find a subject, we insert a word as a
subject that is found by this person estimation feature.
2.6

Question Answering Module

We created four types of question answering modules: precise match, loose match,rough match, and FrameNet modules.
Precise match module extracts civil law articles which proposition clauses matchwith a given proposition clause of problem sentences. For each predicate, we find a
subject and an object by their case markers. We perform exact matches for the predicate, its subject and its object. When we could not find any subject nor any object, weskip that sentence. If a negative expression is detected only in one side, we reverse thecorresponding Yes/No answer. Our system repeats these processes for the number ofthe clauses of the problem sentences. The precise match module has the highest percentage of correct answers among the four modules. However, this module cannot
find answers for about 70 percent of problems.
Loose match module is a looser version of the precise match. When comparing
proposition clauses and condition clauses, this module regards a pair of clauses asmatched if either a pair of objects or a pair of subjects is matched, in addition to the apair of predicates.
For example, if a problem’s clause is { 結 ぶ (sign), 代 理 人 (agent), 契 約(contract)} and if an article’s clause is {結ぶ (sign), 代理人 (agent), 売買契約 (sales
contract)}, then our precise match module judges that they do not have a same meaning because they do not have a same object. On the other hand, our loose match module judges that they have a same meaning as this module ignores objects in this case.
Rough match module is the loosest match in our modules. This module only compares predicates of proposition clauses and then checks whether the predicates include
negative expressions or not.

Furthermore, we prepare the FrameNet module. FrameNet[12] is an English semantical lexical database based on a theory of meaning called frame semantics. The
basic idea is that people understand the meaning of words largely by the frames which
they evoke. Frames have some semantic roles called Frame Elements (FEs) and frame
evoking words are called Lexical Units (LUs). We use LUs of Japanese
FrameNet[13], in addition to the original English version of FrameNet.
We use Japanese WordNet[14], which is a lexical database for Japanese. We use
Japanese WordNet to obtain synonyms, hypernyms, hyponyms and English translation words to expand FrameNet.
The FrameNet module is made by extending the loose match module. The difference of the FrameNet module from the loose match module is in comparing predicates. Using an inheritance relationship of FrameNet, we examine whether a pair ofpredicates represents the same meaning or not. This system measures a distance between nodes of words and regards as the same meaning if the nodes are in a near distance. We defined confidence values for each relationship type to calculate the distance.
In addition, we made more sub-modules by setting different options for these four
modules. An option forces the predicate selection; when a clause includes specificcondition pattern like “の場合は(in case of)” or “の時は(when)”, a previous chunkis regarded as the core predicate of that clause. Another option is in the FrameNet
module; we changed the confidence calculation method and/or the threshold of theconfidence values.
2.7

Module Integration

Fig. 3 Overview of our module integration

After completing the answers for each module, we decide our final answer. We prepared two ways for resolution.
When we do not use SVM, we try applying the precise match module first. Whenthe precise match module cannot be applied, we apply the loose match module. If the9loose match module cannot be applied as well, we try applying the rough match module.
When we use SVM, we use each answer output by each module and the confidence
calculated from the number of articles that matched word sets of the problem sentence
for learning. The confidence is a value that decreases as more the number of the related articles. The equation of calculating the confidence is 1 / n. n is the number of therelated articles when n > 0. The confidence is 0 when n = 0. However, the related
articles of solving the problems are not only one, it is not necessarily true that theprobability that the related articles is less and accurate is higher. The training data for
SVM is the problems of 2006 to 2008.
The optimum cost and gamma value of SVM options were determined by gridsearch. The kernel used is a polynomial kernel. We used a kernel with a high rate ofcorrect answers by comparing correct answer rates for each combination of functions
and modules.

3

Experiment and Result

We applied our system to Task 4 of COLIEE 2018. In Task 4, only the problem sentences were provided without any additional information, asking Yes/No as answers.
3.1

Result of COLIEE 2018
Table 3. Result of COLIEE 2018 formal run

Team

Language

YA
KIS_Frame
KIS_mo3
KIS_dict
KIS_SVMKIS_Frame2
UE

?
J
J
J
J
J
E

# Correct Answers(total 69 answers)
44
39
38
37
36
35
33

Accuracy
0.63880.56520.55070.5362
0.52170.50720.4783

Table 3 shows the result of the COLIEE 2018 Task 4 formal run. Among these results, team names with KIS as their prefix show our results. KIS_Frame uses the
FrameNet module only. KIS_mo3 uses the precise match module, the loose match
module and the rough match module. We selected options of these modules to be thehighest correct answer rate on the training set. KIS_dict uses the same module as
KIS_mo3 while adding our synonym dictionary. KIS_SVM uses SVM for mod-ule
integration; other results integrates modules in the filter based way. KIS_Frame2 uses
the FrameNet module, the precise match module, the loose match module, and therough match module. KIS_Frame was the best result among our submissions for the
COLIEE 2018 formal run.

There is little difference between the results of KIS_dict and KIS_mo3, synonym
dictionary was used or not. Because many problems cannot be answered if the system
do not handle synonyms, the effect of our synonym dictionary would have reduced
due to other issues. As related articles are not given in Task 4, a huge number of combinations are compared by precise match and loose match. Our system does not haveany feature to rank relevant articles. If there are many related articles, even if the correct judgment can be made by a synonym dictionary in the correct article, the result
could be buried. Refining our searching meth-od for the related articles is the future
work. The good result of KIS_Frame was probably because the system was able toeffectively compare the predicates by the frame information.
3.2

Result of past years
Table 4. Result of COLIEE training data (past years’s legal bar exam)
2006

2007

2008

2009

2010

2011

2012

2013

2014

2015

2016

KIS_SVM

0.54

0.57

0.60

0.66

0.51

0.65

0.53

0.62

0.60

0.52

0.53

total
0.57

KIS_mo3

0.57

0.57

0.60

0.60

0.48

0.63

0.51

0.63

0.57

0.52

0.55

0.56

KIS_dict

0.57

0.57

0.52

0.62

0.53

0.61

0.47

0.57

0.57

0.50

0.55

0.55

KIS_Frame

0.48

0.50

0.52

0.60

0.42

0.48

0.55

0.60

0.54

0.48

0.60

0.53

KIS_Frame2

0.51

0.59

0.62

0.58

0.46

0.53

0.44

0.44

0.53

0.50

0.42

0.51

Table 4 shows results of COLIEE training data of past years’ legal bar exam.
KIS_SVM, which uses SVM to mix different modules, performed best in total. However, KIS_SVM is ranked fourth in the formal run, lower than our other runs shown in
the table. Its reason might be fluctuations between different year’s problems. Especially, the problems of this year’s formal run include many concrete stories. Moreover, this year’s formal run includes problems consist of more than two sentences
which were not appeared in the past problems. Because the tendency could have beendifferent for these reasons, training model of SVM might not have fit well with theformal run data.
3.3

Comparison of Results for Difficulty Levels
Table 5. Results for each difficulty level

very easyeasy
difficultvery difficult

Target40
62
44
21

Accuracy
0.75
0.387
0.522
0.380

We manually categorized problems into four difficulty levels. shows results of the
problem in 2014 and 2016. “Very easy” problems were correctly answered by more

11than 70 percent, while “easy” problems and “difficult” problems were less than 40
percent. “Very easy” problems were solved well because our precise match can easilyfind superficially similar expressions in such problems. It was unexpected that “difficult” problems were solved better than “easy” problems. Even in problems that humans feel easy, synonym and zero pronouns resolutions are required, which are very
difficult issues for computers to be solved. Regarding “difficult” problems, a very
complicated structure could be transformed into a simple structure by predicate argument structure analysis, which is easier to solve than the “easy” problems. “Easy”
and “very difficult” were far less than the random baseline (50%).

4Conclusion and Future WorkWe developed an automatic answering system for legal bar exam, which can show
logical structures how the system solved a given legal problem, We focused on answering simpler problems in this paper. We confirmed that our system could correctly
answer more than 70 percent of the simpler problems.
In order to represent structures of legal documents, we implemented a textual entailment solver, which has predicate argument structure analysis as its core part. We
made a variation of different modules using different comparison methods. We used
SVM to make ensemble of these modules, using confidence values calculated fromthe number of related articles.
In addition, we made a legal synonym dictionary. Our synonym dictionary is structured in that a s synonym of a predicate could be conditioned with its object. Unfortunately, we could not achieve a significant difference because there are too many related articles when searching relevant articles. We applied our system to the COLIEE
2018 Task 4 dataset. One of our submission achieved the second rank among participants.
The information retrieval part, which corresponds to Task 3, could be improved toincrease the entire system performance in future..
Acknowledgements
This work was partially supported by MEXT Kakenhi and JST CREST.

References
[1]

[2]

“Competition on Legal Information Extraction/Entailment (COLIEE-14), Workshop
on Juris-informatics (JURISIN) 2014,” 2014. [Online]. Available:
http://webdocs.cs.ualberta.ca/~miyoung2/jurisin_task/index.html.
M.-Y. Kim, R. Goebel, and S. Ken, “COLIEE-2015: Evaluation of Legal Question
Answering,” in Ninth International Workshop on Juris-informatics (JURISIN 2015),
2015.

[3]

[4]
[5]
[6]

[7]

[8]

[9]
[10]
[11]

[12]
[13]
[14]

M. Kim, R. Goebel, Y. Kano, and K. Satoh, “COLIEE-2016 : Evaluation of theCompetition on Legal Information Extraction and Entailment 2 . The Legal Question
Answering Task,” 2016.
Y. Kano, M. Kim, R. Goebel, and K. Satoh, “Overview of COLIEE 2017,” vol. 47, no.
Icail, pp. 1–8, 2017.
D. Lin and P. Pantel, “Discovery of Inference Rules for Question Answering,” Nat.
Lang. Eng., vol. 7, no. 4, pp. 343–360, 2001.
D. Pinto, A. McCallum, X. Wei, and W. B. Croft, “Table Extraction Using
Conditional Random Fields,” Proc. 26th Annu. Int. ACM SIGIR Conf. Res. Dev.
informaion Retr. - SIGIR ’03, p. 235, 2003.
D. Ravichandran and E. Hovy, “Learning Surface Text Oatterns for a Question
Answering System,” Proc. 40th Annu. Meet. Assoc. Comput. Linguist. - ACL ’02, no.
July, p. 41, 2001.
H. Yu and V. Hatzivassiloglou, “Towards Answering Opinion Questions: Separating
Facts from Opinions and Identifying the Polarity of Opinion Sentences,” Proc. 2003
Conf. Empir. methods Nat. Lang. Process., p. 129–136., 2003.
D. A. Ferrucci, “Introduction to ‘This is Watson,’” IBM J. Res. Dev., vol. 56, no. 3.4,
p. 1:1-1:15, 2012.
D. A. Ferrucci, “IBM’s Watson/DeepQA,” SIGARCH Comput. Arch. News, vol. 39(3),no. Journal Article. doi:10.1145/2024723.2019525, 2011.
S. Polsley, P. Jhunjhunwala, and R. Huang, “CaseSummarizer: A System for
Automated Summarization of Legal Texts,” Proc. COLING 2016, 26th Int. Conf.
Comput. Linguist. Syst. Demonstr., pp. 258–262, 2016.
“Welcome to FrameNet!” [Online]. Available:
https://framenet.icsi.berkeley.edu/fndrupal/.
“Japanese FrameNet.” [Online]. Available: http://jfn.st.hc.keio.ac.jp.
“Japanese WordNet.” [Online]. Available:
http://compling.hss.ntu.edu.sg/wnja/index.en.html.