Overview of COLIEE 2024 Japanese Statute Law Tasks
Task 3: The Statute Law Retrieval Task

Masaharu Yoshioka
Hokkaido University

Task settings
nTarget documents– Japanese Civil law articles are provided in two languages(Japanese and English: 768 articles in total)
Data is updated because of the update of Japanese CivilCode.

nQuestions and gold standard data
– Questions are selected from Japanese Bar exam in twolanguages (Japanese and English)• Training 1,097 questions (extracted from exam
conducted in 2006-2022 exam)• Test: 109 questions (extracted from exam conducted in2023 exam)*– Gold standard data
• Relevant articles and entailment results (true/false) are
judged by the person in legal domain

Regulation for using External Resourcesn Concern about using GPT-4 at COLIEE 2023

– Reproducibility problem– Contamination problem
n New regulation– Reproducibility
External resources used for the submission should beavailable for other participants.
– Contamination
If the external resource (such as LLM) is machinelearning based one, training data for the resource should
be disclosed.

Evaluation Format
n Evaluation measure– Usage of macro average (calculate average over all
evaluation measures for each question)– Usage of F2 (put more emphasis on recall)
It is important to provide all related articles for theentailment process.
5 x Precision x Recall
F2−measure =4 x Precision + Recall
n Submission of long ranked list (top 100)

– We can analyze type of difficult question

Submitted Runsn 20 runs from 8 teams– LLM prompt
1st• AMHR(3 runs), CAPTAIN(3 runs), JNLP(3 runs)– Deep Neural LM (e.g., BERT, T5) or sentence
embeddings
3rd2nd• BM24 (1 run) NOWJ(3 runs), TQM (3 runs), UA (3runs)– No information• PSI
JNLP and BM24 uses LLMs that are trained using undisclosed data

Evaluation results basedn Selected list for the best run of each teamTeam

returnretrieved

F2Precision

RecallJNLP *

188

99

0.807

0.709

0.870 0.801

CAPTAIN

168

96

0.800

0.732

0.845 0.815

TQM

14089

0.782

0.785

0.800 0.790

NOWJ

202

96

0.772

0.690

0.835 0.756

AMHR

185

950.749

0.651

0.825 0.740

UA23391

0.711

0.610

0.800 0.700BM24 *425

94

0.539

0.282

0.795

-

PSI

109

90.086

0.090

0.085

0.2

JNLP and BM24 uses LLMs that are trained using undisclosed data

MAP

R05-26-I

0F2
precision
recallMAP
R@5
R@30
R05-20-E

R05-19-O

R05-10-O

R05-06-O

R05-06-A

R05-11-I

R05-10-U

R05-24-O

R05-27-A

R05-23-A

R05-10-A

R05-11-U

R05-12-U

R05-05-I

R05-36-O

R05-26-A

R05-05-E

R05-06-E

R05-15-O

R05-12-I

R05-18-U

R05-09-U

R05-36-U

R05-02-A

R05-02-I

R05-21-O

R05-21-I

R05-09-A

R05-23-I

R05-06-I

R05-21-E

R05-18-O

R05-15-A

R05-25-U

R05-18-E

R05-04-E

R05-18-A

R05-09-E

Average Retrieval Results for Single Answer

n 88 questions– Easy(Average of F2 is larger than 0.8): 49– Difficult(Average of F2 is lower than 0.2): 13
Average performance for the difficult questions(Average F2 is lower than 0.8)

1

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

Average Retrieval Results that have 2Relevant Articles
n 21 questions: Recall is are comparatively lower

than ones for single question– However, performance is comparatively better than
COLIEE 2023
10.9
0.8
0.7
0.6
0.5
0.4
0.3
0.20.1

recallR@5-O-U

R0514-E

512

R0-A

507

R0505-E

R@30R0-E

526
R0-O

513

R0-I
R0526-I

536

MAP

R0-I

518
R0-U

524

R0-I

513

R0510-O

precision

R0-U

513
R0-I

502

R0-I

519

F2R0-E

509

R0-I

525

R0-U

R0525-A

527

R0519

R0R0511-E

0Difficult questionsn 15 questions (average F2 is lower than 0.2)– 9 questions out of 15 use anonymized symbol (such asA and B)• Question R05-6-A
If A concludes a gift agreement with B to the effectthat A will give B 100,000 yen if B does not commit
theft for a certain period of time, and if B, who has
not committed theft for that period of time, demands A
to pay 100,000 yen, A may refuse to do so.

• Relevant article(Unlawful Conditions)
Article 132 A juridical act subject to an unlawful
condition is void. The same applies to a juridical act
subject to the condition that an unlawful act not beperformed.

Performance for the anonymized question(45 questions)n AMHR, JNLP, CAPTAIN that use LLM prompt

have better performance than others (TQM is thebest among others)– LLM prompt may be good to handle semantic matching.
Team

returnretrieved

F2Precision

Recall

MAP

AMHR

87

420.669

0.561

0.756 0.726

JNLP

83

39

0.662

0.586

0.722 0.735

CAPTAIN95

420.6470.497

0.756 0.742

TQM

5633

0.628

0.678

0.633 0.719

Summaryn There are many questions that can be retrieved bythe all systems.
– We still have problems to retrieve questions withanonymized symbols.
– However, LLM prompt has slightly better performance
than previous methods.