Overview of Benchmark Datasets andMethods
for the Legal Information
Extraction/Entailment
Competition (COLIEE) 2024
Randy Goebel, Yoshinobu Kano, Mi-Young Kim,
Juliano Rabelo, Ken Satoh, Masaharu Yoshioka,

Task 2
Mi-Young Kim, University of Alberta

1

Task 2 Description• Legal Case Entailment Task
• Involves the identification of a paragraphfrom existing cases that entails thedecision of a new case
• Given a decision Q of a new case and a
relevant case R, a specific paragraph in R
that entails the decision Q needs to beidentified.
• The answer paragraph cannot be identified
merely by information retrieval techniques– Because the case R is a relevant case to Q,
many paragraphs in R can be relevant to Q
regardless of entailment.

2Task 2 Example

Base case

B232 arrived in Canada with 491
other persons aboard the MV Sun
Sea...

Decision of the base case

Given that the Respondent remains
a security risk whom the Minister
has...

Noticed (relevant previous) case[P1] Previous decisions to detain theindividual must be...
[P2] The Ministers are requesting anorder...
…
[P39] THIS COURT ORDERS thatthe stay motion be granted until thefinal ...

Entailing paragraph to the decisionof the base case

P273

Evaluation Measure and Dataset• F1-measure : harmonic mean of precision and recall• The training data : 725 query cases and 25,783
paragraphs were provided for training.
• The test data: 100 query cases and 3,651 paragraphs• The data is drawn from an existing collection of
predominantly Federal Court of Canada case law.
• Training data consists of triples of a query, a noticed
case, and a paragraph number of the noticed case bywhich the decision of the query is entailed. Here,’noticed case’ means the relevant case of the query.
• Test data does not include the paragraph number ofthe noticed case. The goal of Task 2 is to identify
this paragraph number.

4Participation in Task 2• Six teams (total 18 submissions)– 3 submissions per team
(Each team was allowed maximum threesubmissions.)5Submitted methods
TeamApproaches

AMHR [8] (threeruns)

proposed two approaches: (1) finetuning a legal-BERT model withtriplet loss with labels as positive examples and all otherparagraphs as negative examples on the train set provided fortask 2. This approach resulted in overfitting. (2) finetuning a
monoT5 model pre-trained on the MSMARCO dataset with hard
negative mining examples chosen by BM25 and another versionof the monoT5 model itself. They choose the top-2 predictions by
this model as long as the ratio between their similarity score isless than 6.619 (a hyperparameter found by grid search);otherwise, they choose just the first prediction. The second
approach got the best results on task 2, this year.

CAPTAIN [6](three runs)

introduces a method that builds upon the state-of-the-art
approach used in Task 2 of the 2023 competition. This method
incorporates zero-shot and few-shot learning techniques toleverage the knowledge stored in large language models. Initially,
they fine-tune a pre-trained monoT5 sequence-to-sequencemodel using hard negative sampling to produce an output. For
each query paragraph, they select the top-k candidates with thehighest scores to create zero-shot and few-shot prompting
techniques for in-context learning with FlanT5 LLM.
6[5]Nguyen, C., Tran, T., Le, K., Nguyen, H., Do, T., Pham, T., Luu, S.T., Vo,T., , Nguyen, L.M.: Pushing the Boundaries of Legal Information
Processing with Integration of Large Language Models. In: Proceedingsof the Eighteenth International Workshop on Juris-informatics(JURISIN2024) (2024)[6]

Nguyen, P., Nguyen, C., Nguyen, H., Nguyen, M., Trieu, A., Nguyen, D.,
Nguyen, M.: CAPTAIN at COLIEE 2024: Large Language Model for Legal
Text Retrieval and Entailment. In: Proceedings of the EighteenthInternational Workshop on Juris-informatics(JURISIN 2024) (2024)[7]

Nguyen, T.M., Nguyen, H.L., Nguyen, D.Q., Nguyen, H.T., Vuong, T.H.Y.,
Nguyen, H.T.: NOWJ@COLIEE 2024: Leveraging Advanced Deep
Learning Techniques for Efficient and Effective Legal Information
Processing. In: Proceedings of the Eighteenth International Workshop on
Juris-informatics(JURISIN 2024) (2024)[8]

Licato, J.: AMHR COLIEE 2024 Entry: Legal Entailment and Retrieval. In:
Proceedings of the Eighteenth International Workshop on Jurisinformatics(JURISIN 2024) (2024)[11]

Wehnert, S., Murugadas, V., Naik, P.V., Luca, E.W.D.: Improving
Robustness in Language Models for Legal Textual Entailment throughArtifact-Aware Training. In: Proceedings of the Eighteenth International
Workshop on Juris-informatics(JURISIN 2024) (2024)

7

Submitted methods
TeamApproaches

JNLP [5] (threeruns)

fine-tuned MonoT5 on the training set of Task 2 with hard
negative sampling. The model MonoT5 is a T5-3B
reranker finetuned on the MS MARCO passage dataset
for 10k steps. They used Flan-T5 and Mixtral forprompting.

NOWJ [7] (three proposes two approaches of entailment recognition,runs)
using multilingual BERT and monoT5 for the three runs.
MonoT5 is a T5-based re-ranking model fine-tuned for thedownstream task of classification, while mBERT is a
traditional approach for document re-ranker.
Multilingual BERT and training the mBERT model withweak labels [10] were their last year’s solutions.
Therefore, for the first two runs, they finetuned the models
on this year’s dataset.

8Submitted methods
TeamApproaches

OVGU [11] (threeruns)

team’s proposed approach involves using a chain of pre-trained Custom
Legal-BERT models that are fine-tuned on sub-datasets generated using
BM25 and a Bi-Encoder to select the top-N candidate paragraphs. To
enhance the models’ robustness, a binomial test is employed for artifact
detection. OpenAI’s GPT-3.5-turbo model is used to create adversarial
instances for selected training instances with annotation artifacts. The large
language model was prompted to switch the previous negative entailment
label into a positive one for balancing out the training examples withannotation artifacts. These instances, along with the top-N candidate
paragraph dataset, are further used to fine-tune the models. A chained
approach is applied during prediction: If the first model (specialized for highprecision) fails to predict a hypothesis with at least one premise as’Entailed,‘ the second model is used for that hypothesis. If any hypotheses
are missed after using the second model, the BM25 top-ranked premisefound for a given hypothesis is labeled as ’Entailed.‘

9Task 2 official results
TeamF1AMHR

Precision

RecallTeam

0.6512 0.6364

0.6667

JNLP

0.6320 0.6967CAPTAIN

F1Precision

RecallCAPTAIN 0.6360

0.72810.5646

0.5782

CAPTAIN 0.6235

0.77000.52380.6235 0.7700

0.5238NOWJ

0.61170.61810.6054

JNLP

0.6045 0.6694

0.5510OVGU

0.5962

0.5636

0.6327NOWJ

0.5946 0.5906

0.5986

JNLP

0.5912

0.63780.5510OVGU

0.5705 0.5506

0.5918OVGU

0.55320.50000.6190NOWJ

0.5197 0.5032

0.5374MIG

0.47010.56730.4014MIG

0.4696 0.5800

0.3946AMHR

0.35420.36170.3469AMHR0.3320 0.4100

0.2789MIG

0.1364

0.09790.2245
10

Number of the answer paragraphs<2024>
Number of the answer paragraphs

123

4

Number of queries

6525

82

Number of the answer paragraphs

1234

Number of queries

86

94

1

<2023>

12Discussion• The AMHR team attained the best results.
• CAPTAIN used last year’s winner model,
which is based on a fine-tuned monoT5,
and their model was ranked second.
• The first ranked model also used finetuned monoT5, but they used ahyperparameter value as a threshold of
the similarity score, and got the best
result this year.

13Certificatefor Task 2 winnerCompetition
on Legal Information Extraction/Entailment(COLIEE) 2024
Sponsored by Alberta Machine Intelligence Institute (AMII)
University of Alberta
National Institute of Informatics (NII)

Team name: AMHR
Affiliation: University of South Florida, USA
Your team achieved the highest performance on Task 2 of
the COLIEE Competition.
Sincere thanks for your contribution to the growing
community of research scholars who have invested theirenergy and talent into pushing the boundaries of researchand its application to Juris-Informatics.
May 29th, 2024
COLIEE organizers,
Randy Goebel
Mi-Young Kim
Juliano Rabelo
University of Alberta,
CanadaYoshinobu Kano,
Shizuoka University,Japan
Masaharu Yoshioka,
Hokkaido University,Japan

Ken Satoh,
National Institute ofInformatics (NII),
Japan

14