Summary of the Competition on Legal
Information, Extraction/Entailment(COLIEE) 2024
Randy Goebel, Yoshinobu Kano, Mi-Young Kim,
Juliano Rabelo, Ken Satoh, Masaharu Yoshioka

COLIEE•

11th edition•

Initial editions had two tasks on Statute Law•

Since 2018, two tasks on Case Law have been added

Task 1 - Description● Case Law Retrieval● Competitors are given a set of cases and a mapping containing query

cases and their respective noticed cases● The test dataset does not contain noticed cases, and the task is topredict which cases from the dataset should be noticed with respect
to each query case
● Submissions are evaluated by the F1-measure (harmonic mean ofprecision and recall)Task 1 - Dataset• The data is drawn from a collection predominantly comprised of cases
from the Supreme Court of Canada• Training data: 5,616 cases, of which 1,278 were used as query cases• Test data: 1,734 cases, of which 400 were used as query cases• Direct citations are removed from the case contents• Training data consists of a list of query cases and their respective
noticed cases• Test data does not include the list of noticed cases for each querycase. The challenge in Task 1 is to predict those noticed cases

Task 1 - Training Dataset

Task 1 - Training Dataset

Task 1 - Training Dataset

Task 1 - Training Dataset

Task 1 - Training Dataset

Task 1 - Participation• Ten teams (total 26 submissions)• Each team could send at most three submissions• Eight teams sent papers describing their methods

Task 1 - Official ResultsTask 1 - Methods
TeamDescription

TQM (winningteam)

Used lexical matching and dense vector retrieval to generate features (plus some simple features such as caselength) that were submitted to a learning to rank method. The authors also applied pre and post processing toavoid irrelevant information. Their method not only applies all of those techniques, but aims at a deeper
understanding of the case trying to capture the main facts described in the case.

JNLP

Proposed a three-phase approach: the first stage performs retrieval after splitting the query document intoparagraphs and using a BM25 model with top-k cutout to retrieve candidate documents. Phase two is a
re-ranking stage. The last stage is where prediction actually happens: after the re-ranking stage, for each query
document, the authors select the top-k candidate documents from the re-ranked list as prediction with kselected, using grid-search on the validation set. They also developed an ensemble strategy by concatenating
the prediction results of the re-rankers before selecting the top-k to boost the recall metric of the system.

BM24

Organize each case into segments summarized by gpt-3.5. Among them, one segment is selected to represent
the case. An embedding of that segment is stored in FAISS. A segment of the query case is used to query theFAISS vector store to retrieve similar cases. AnglE is used as the sentence embedding model, trained from an
open source model on the Task 1 training set pre-processed in the same way.

CAPTAIN

Performed some heuristic pre-processing steps, then used TF-IDF and BM25 to extract keywords and retrieve
relevant documents, and then applied LLMs to summarize the decisions and perform fine-tuning of a retrievalmodel based on such summaries

Task 1 - Methods
TeamDescriptionNOWJDeveloped an approach based on a combination of BM25 and a pre-trained Longformer. After an initial
pre-processing step, BM25 is used to calculate the similarity between each pair of query case and candidate case.
The result is used as a pre-ranking input to the Long-Former model. Scores from BM25 and LongFormer are then
combined, with parameters being defined after a grid search is conducted.

MIG

Developed an informative baseline for Task 1 that does not apply any LLMs. The authors vectorized the cases with atool on BERT-base and BERT-large. After vectorizing the cases, they compute the cosine similarity between thecandidate cases and the given new case using FAISS. Then, for a new case, the authors ranked the candidate cases by
their cosine similarity with the new case, and chose 20 candidates that were most similar to the new case. Then thedifference between the cosine similarity between the i-th most similar case and the (i + 1)-th most similar case (di) iscalculated, and the first i cases are recommended if di > 2d1

UBCS

Applied TF-IDF to rank cases varying how the model is used. Their first approach is a baseline, with vanilla TF-IDF
weighting model being used to retrieve and rank noticed cases for each given query case. The second approach
applies summarization only on the query cases before using TF-IDF for retrieval. The third approach applies
summarization for both the query and candidate cases.

UMNLP

Developed a pairwise similarity ranking framework. The authors train a feed-forward neural network to perform abinary classification task based on several features from each query-candidate case pair. Those features include theextraction and similarity matching for a novel feature which the authors call a “proposition” (a short summary of thebasis upon which a noticed case has been cited), as well as the name of the judge deciding the case, verbatim
quotations from the text, and several other novel features

Task 1 - Discussion• Scores improved steadily since the new format for this task wasintroduced. Compared to last year, almost a 50% increase (0.30 to0.44)• Most competitors applied some transformer-based approach (at least
as a component of their method)• Usually transformer-based methods are combined with traditional IR
methods such as BM25, and augmented with heuristics

Task 1 - WinnerCertificate

Task 1 - Summary• Legal Case Retrieval Task
• Requires identification of noticed cases with respect to a query case
• The answer cannot rely on simple pattern matching as the explicit
case citations are removed from the case contents• Ten teams and 26 submissions• Approaches predominantly relied on some combination oftransformer-based and traditional IR methodsSummary of the Competition on Legal
Information, Extraction/Entailment(COLIEE) 2024
Randy Goebel, Yoshinobu Kano, Mi-Young Kim,
Juliano Rabelo, Ken Satoh, Masaharu Yoshioka