Task 4: Textual Entailment inStatute Law
COLIEE 2024 Overview @ JURISIN 2024
狩 野芳 伸Yoshinobu Kano
静岡大学 情報学部
行 動 情 報 学 科

1

COLIEE Competition￭ Competition for Legal Information Extraction
and Entailment (COLIEE)◨ I am one of the organizers◨ COLIEE 2013, 2014, 2015, 2016, 2018, 2020,2022, 2024 in JURISIN◨ COLIEE 2017, 2019, 2021, 2023 in ICAIL￭ Case Law tasks (from 2018)◨ Canadian Federal Court database￭ Statute Law tasks◨ Japanese Legal bar exam
◨ Human applicants should pass the Legal Bar
Exam to be a lawyer in JapanYoshinobu Kano

Example of Statute Task
Question A special provision that releases warranty can be made,but in that situation, when there are rights that theseller establishes on his/her own for a third party, theseller is not released of warranty.
RelatedArticle(TaskIR)(Special Agreement Disclaiming Warranty) Article 572
Even if the seller makes a special agreement to theeffect that the seller will not provide the warranties set
forth from Article 560 through to the preceding Article,
the seller may not be released from that responsibility
with respect to any fact that the seller knew but did
not disclose, and with respect to any right that theseller himself/herself created for or assigned to a thirdparty.

Label(TaskTE/QA)

Yes3

COLIEE 2024 Task 4 Dataset￭ Dataset same as Task 3￭ COLIEE 2024 training data

◨ 1097 queries◨ built from the bar exam (short answer test) civil code part
◨ published in 2006-2023◨ XML files, each corresponds to one year’s publication￭ Japanese Civil Law Articles as knowledge base
￭ Both in original Japanese version and manuallytranslated English version￭ COLIEE 2024 test data
◨ 109 queries from the latest bar exam of 2023

￭ Each team can submit up to three runs for each task

◨ We asked to submit past formal run configurations as well
- 2021 (R02), 2020 (R01), 2019 (H30)4Overview: Historical Development￭ Linguistic structures specific to legal docs
◨ ～COLIEE 2019: classic NLP￭ Insufficient data size (pretrain/finetune)◨ ～COLIEE 2020: deep language model bytransfer learning (pretraining)◨ ～COLIEE 2022: ensemble of different systemoutputs◨ COLIEE 2023～: LLM, generative AI￭ General knowledge, evidence/explanations◨ ???: common sense, relationships, logic, etc.
◨ COLIEE 2025 – new task/evaluation planned!

Clauses and Predicate Arguments
Proposition Clause

H24-2-1

Conditional Clause

制限行為能力者のした契約について，制限行為能力者及びその法定代理人が取消権を有するときは，契約の相手方も取消権を有する。
An act which may be rescinded on the grounds of the limited capacity to act ofthe person who performed such act may be rescinded only by the person
whose capacity to act is limited, or its agent, successor, or a person who hasthe authority to give consent.
Predicate

Subject
契約の相手方も取消権を有する。
ObjectSubjectPredicate

制限行為能力者及びその法定代理人が取消権を有するときは,
ObjectPAS in proposition clause{有する, 相手方, 取消権}
(has, the counterparty, the rightto rescind )PAS in conditional clause:{有する, 法定代理人, 取消権}
(have, the statutory agent, theright to rescind )Overview: Historical Development￭ Linguistic structures specific to legal docs
◨ ～COLIEE 2019: classic NLP￭ Insufficient data size (pretrain/finetune)◨ ～COLIEE 2020: deep language model bytransfer learning (pretraining)◨ ～COLIEE 2022: ensemble of different systemoutputs◨ COLIEE 2023～: LLM, generative AI￭ General knowledge, evidence/explanations◨ ???: common sense, relationships, logic, etc.
◨ COLIEE 2025 – new task/evaluation planned!

Yoshinobu Kano

Results￭ Total 8 teams (24runs)￭ CAPTAIN2 winner!

◨ LLM (flan-T5) withdata augmentationand heuristic rules,fine-tune.

￭ *JNLP 2nd◨ different LLMs
(Wqen (their originalmodel), Mistral, FlanAlpaca, and FlanT5)
ensemble the resultswith majority voting,took the top-1
prompt from FlanAlpaca
9Results￭ UA_stack 3rd◨ used zero-shot learningon google/flant5-xxl
with PromptSource8 forfinding potential good
prompts, added
positive and one
negative example,
chose the top 3
prompts, finally zeroshot inference with all
three prompts andvoting between them.

￭ ∗ indicates runs using
not fully disclosedmodels￭ + indicates runs withpreprocessing by suchmodels10

Task 4 Results (Textual Entailment)￭ Comparison with previous
formal run settings(training/eval)◨ 2021 (R02), 2020 (R01), 2019(H30)◨ asked to apply with this year’ssame system￭ Different year shows quite
different charcteristics due tothe datasets◨ any way to get more stableresults?

11

Open Questions and Future
Plans: What are the LLMs doing?

￭ LLMs could answer quite accurately◨ evidences are also fine in most cases◨ LLMs (implicitly) includes answers similar to our gold
data in their training?
◨ Humans can perform “symbolic processings” “logicalcalculations”◨ but LLMs should not perform “logical calculation” rather“compositions”◨ Needs precise analysis regarding what sort of issues
are actually solved￭ Can LLMs “logically” think?
￭ “Explainable AI” required in two meanings◨ explanation for humans◨ explanation of the internal process
Yoshinobu Kano

Difficult Examples￭ Temporal expressions, coreferences◨ If person A donates a house that he/she isrenting to person C with a provision for thepayment of rent at the end of every month toperson B midway through the month, if there
is a special agreement between person A andperson B, the rent for the month will be
distributed between person A and person B in
proportion to the number of the days.

￭ Document structure, references, Negation(sometimes implicit in terms), Acronyms(domain dependent, vague), …
Yoshinobu Kano

Overview: Historical Development￭ Linguistic structures specific to legal docs
◨ ～COLIEE 2019: classic NLP￭ Insufficient data size (pretrain/finetune)◨ ～COLIEE 2020: deep language model bytransfer learning (pretraining)◨ ～COLIEE 2022: ensemble of different system
outputs◨ COLIEE 2023: LLM?

￭ General knowledge, evidence/explanations◨ ???: common sense, relationships, logic, etc.
◨ COLIEE 2024 – new task/evaluation planned!

Look forward to see new participantsin COLIEE 2025!