言語処理学会 第30回年次大会 発表論文集（2024年3月）

GPTs and Language Barrier: A Cross-Lingual
Legal QA Examination
Nguyen Ha Thanh1 Hiroaki Yamada2 Ken Satoh1
1 National Institute of Informatics
2 Tokyo Institute of Technology{nguyenhathanh,ksatoh}@nii.ac.jp yamada@c.titech.ac.jp

Abstract
In this paper, we explore the application of Generative Pre-trained Transformers (GPTs) in cross-lingual legal Question-Answering (QA) systems using the COLIEE
Task 4 dataset. In the COLIEE Task 4, given a statement
and a set of related legal articles that serve as context, theobjective is to determine whether the statement is legallyvalid, i.e., if it can be inferred from the provided contextual articles or not, which is also known as an entailment
task. By benchmarking four diﬀerent combinations of English and Japanese prompts and data, we provide valuable
insights into GPTs’ performance in multilingual legal QA
scenarios, contributing to the development of more eﬃcient and accurate cross-lingual QA solutions in the legal
domain.

1

Introduction

The rapid increase in cross-border transactions and theglobalization of legal systems have highlighted the importance of eﬀective legal information retrieval across various languages. Legal Question-Answering (QA) systems
assist professionals and individuals in locating relevant legal resources and extracting information to resolve disputes, comply with regulations, or make informed decisions. However, the vast amount of legal information
available in diverse languages poses signiﬁcant challenges
to the development of eﬃcient and accurate cross-lingual
QA systems.
Current legal QA systems predominantly focus on monolingual information retrieval, which limits their capacity toaddress the needs of an increasingly multilingual and multicultural user base. Developing cross-lingual legal QA
systems necessitates a deep understanding of not only thelinguistic nuances but also the cultural and legal diﬀerences
between the source and target languages. This requires advanced natural language processing techniques and robustmodels to accurately and eﬀectively retrieve relevant information across languages.
In this study, we aim to investigate the potential of Generative Pre-trained Transformers (GPTs) in addressing theaforementioned challenges in cross-lingual legal QA systems. We utilize the COLIEE Task 4 dataset, which isderived from the Japanese Bar Exam and originally inJapanese, with the English version professionally translated by legal experts. The dataset comprises legal QA
data related to entailment tasks in both languages. Task 4involves determining the legal validity of a given statement
based on a set of contextual legal articles. We evaluate
the performance of GPTs under various settings, including
diﬀerent combinations of prompts and data across theselanguages, to gain insights into their eﬀectiveness in crosslingual legal QA scenarios.
Our research contributes to the broader understandingof GPTs’ capabilities and limitations in cross-lingual legal
QA scenarios, potentially paving the way for the development of more eﬃcient and accurate multilingual legal QA
systems. It also contributes to bridging the language gap
in the legal domain, ultimately making legal information
more accessible and comprehensible for users, regardlessof their linguistic background.

2Related Work

Research in the legal domain has presented numerous
challenges for the Natural Language Processing (NLP)
community, primarily due to the complexity of legal language and the need for contextual understanding in legal
reasoning. As such, numerous studies have been conducted― 1062 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

to improve the capacity of NLP models for legal tasks.

Year Question Count

Legal textual entailment, a prominent area of research,
has utilized the COLIEE Task 4 dataset [1, 2] as a crucial benchmark. Various methods have been employed totackle this task, including traditional NLP techniques suchas keyword-based matching [3] and rule-based systems[4] for information extraction and entailment identiﬁcation
Table1from legal documents. However, these approaches may

H2958

H30
R0170111

R02

81

R03109Number of questions for each year in the dataset.

suﬀer from rigidity in methodology and limited ﬂexibility
in responding to subtle changes in language patterns andcontext.

3Experiment Design
In this section, we detail the experimental settings used

With the advent of word embeddings like Word2Vec [5]to evaluate the performance of GPT-4 and GPT-3.5 models

and transformers, such as BERT [6], RoBERTa [7], andin monolingual and cross-lingual scenarios. We utilize theALBERT [8], NLP made signiﬁcant advancements, lead-COLIEE Task 4 dataset, which provides legal questions

ing to improved performance across various legal tasks. Asa result, there have been studies on diﬀerent tasks like legal

and answers in both English and Japanese.

lawfulness classiﬁcation [9], information extraction [10],3.1

question-answering [11], and multi-task learning [12] thatleverage these techniques.

Data Analysis

The dataset comprises data from diﬀerent years: H29,Few-shot learning and few-relational learning strategies

H30, R01, R02, and R03, with a varying number of questions each year, as displayed in Table 1.

have also been introduced in legal entailment tasks [13, 14].

Further, we present the length statistics of the dataset

With these approaches, models can learn from a limited

in English and Japanese languages in Tables 2 and 3, re-amount of labeled data to generalize for unseen cases,

spectively. The statistics provide insights into the dataset’s

which is invaluable given the scarcity of high-quality, an-characteristics, giving us a better understanding of the tex-notated legal documents.

tual complexity and sizing in each language.

Recent GPT models have shown impressive performancein various NLP tasks, including statutory reasoning tasksYear[15, 13, 16], further demonstrating their applicability in theContext Length

Question Length

MinAvgMaxMinAvg

Max

legal domain. GPT-4, speciﬁcally, has delivered remarkable results on the Uniform Bar Examination (UBE) [17],

H29

118

655.312505

60

273.86

601

H30

113

525.101682

40207.24847

showcasing the potential of GPT models in various legal

R01
R02

109
99536.04 2604
541.62 1880

4460

200.49215.78

501
379

R03

68

702.9966

234.62558

reasoning tasks [18, 19].
Building upon these advancements, our study aims toevaluate the performance of GPT-3.5 and GPT-4 on theTable2

legal textual entailment task using the COLIEE dataset, focusing on both monolingual and cross-lingual aspects. We
analyze the models’ performance across diﬀerent dataset4048

English dataset character length statistics.

The data indicate ranges of average lengths for context
in English and Japanese as follows:

versions, ranging from Heisei 18 (2006) to Reiwa 3 (2021),
to gain a better understanding of the potential challenges• English context length varies from 525 characters(H30) to 703 characters (R03).

and capabilities of these state-of-the-art models in the com-

• Japanese context length ranges from 110 charactersplex legal domain. This in-depth analysis, along with the(H30) to 213 characters (R03).

exploration of cross-lingual and global law aspects, provides valuable insights into the prospects of employing
GPT models in legal QA systems for various languages
and jurisdictions.

English question average lengths ﬂuctuate between 200
characters (R01) and 273 characters (H29), while Japanese
question lengths range from 72 characters (H30) to 88 characters (R02). Analyzing these variations in average lengths― 1063 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Year

Context Length
MinAvgMaxMinAvg

Max

H2937

165.745662887.16

198

H30

18

110.46409

1972.84

285

R01

39

168.39

728

15

73.20

223

R02

37

167.57

536

21

88.56

219

R03

29

213.28

1145

29

77.88

173

Table3to assess how well GPT-3.5 (ChatGPT) and GPT-4 can han-Question Length

dle linguistic nuances, cultural diﬀerences, and mappings
between languages in the legal domain. This information
will provide valuable insights into their potential for deployment in multilingual legal QA systems.
With a focus on GPT-4 and GPT-3.5 models, our experiments aim to examine their performance under thesediverse conditions, providing valuable insights into theirJapanese dataset character length statistics.

potential for deployment in multilingual legal QA systems.

of both context and question in diﬀerent years is important

4Experimental Results

since it provides insights into the dataset’s complexity andIn Table 4 and Figure 1, we compare the performance ofpotential text processing challenges.
Looking at Tables 2 and 3, we can see that the maximum

GPT-4 (gpt-4-0314) and GPT-3.5 (gpt-3.5-turbo) in mono-

length of context is 4,048 characters and that of questions
is 847 characters, which is considerably less than the token

lingual and cross-lingual settings. The temperature is set
at 1, Top P is set at 1, the frequency penalty is 0, and thelimit of GPT-4 (0314) at 8,192 tokens and GPT-3.5 (Turbo)presence penalty is 0 for both models. While it is important

at 4,191 tokens. Additionally, it should be noted that one

to note that the years H29 to R03 are independent and nottoken may contain one or more characters1）.

related, we can still observe certain trends and patterns in

3.2

the results.

Monolingual
PromptingandCross-lingual

It is evident that the GPT-4 model consistently outper-

We formatted the input prompt as follows for monolin-

forms the GPT-3.5 model across all independent yearlygual prompting:instances in both monolingual and cross-lingual settings.

Prompt in English:

This superior performance could be attributed to the ad-{context}

vancements in the model architecture and higher capacity,

Question: {question}

allowing GPT-4 to capture the complexities of the lan-Answer (Y or N), no explain.

guages and tasks more eﬀectively.
Furthermore, monolingual settings generally yield

Prompt in Japanese:{context}
質問: {question}higher accuracy scores than cross-lingual settings for bothmodels. This can be attributed to the fact that monolin-

回答 (Y または N)、説明は不要。

gual settings tend to be more straightforward, not requiring

Here, context represents the relevant articles, andtranslation within the same language. On the other hand,question corresponds to a given question in the dataset.

cross-lingual settings involve handling linguistic nuances,

This prompt format ensures consistency in the input andcultural diﬀerences, and complex mappings between lan-allows the models to focus solely on providing a binaryguages, making the task more challenging for even ad-answer―either “Y” (YES) or “N” (NO).

vanced models like GPT-4 and GPT-3.5.

In our experiments, we explore diﬀerent combinations

The observation that Japanese monolingual performanceof context and question languages, yielding four distinct

is better than English monolingual performance can also

settings: English context and English question (EN-EN),be explained by the nature of the original data being inJapanese context and Japanese question (JA-JA), and twoJapanese. Given that the source data is in Japanese, it iscross-lingual settings: English context with Japanese ques-reasonable to infer that the models may have been moretion (EN-JA) and Japanese context with English question(JA-EN).

eﬀective in understanding and processing the original datain Japanese text. The derived English translations primar-

By evaluating the models in both monolingual (EN-EN,ily serve as reference material and might not capture theJA-JA) and cross-lingual (EN-JA, JA-EN) settings, we aim

ﬁne-grained nuances as eﬀectively as the original Japanese1） https://github.com/openai/tiktoken

text, contributing to the observed diﬀerence in perfor-― 1064 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

YearsTable4Model

H29

H30

R01

R02

R03

GPT-4 JA-EN (Cross-Lingual)

0.72410.68570.76580.79010.8349

GPT-4 EN-JA (Cross-Lingual)

0.67240.75710.81980.79010.7798

GPT-3.5 JA-EN (Cross-Lingual)0.60340.65710.5946

0.64200.6697

GPT-3.5 EN-JA (Cross-Lingual)0.56900.71430.69370.60490.6514

GPT-3.5 EN (Monolingual)0.62070.58570.6486

0.69140.7156

GPT-4 EN (Monolingual)

0.7414 0.7429

0.81080.81480.8440

GPT-3.5 JA (Monolingual)0.6207 0.6714

0.56760.5926

0.6514

GPT-4 JA (Monolingual)0.7586 0.7857

0.78380.86420.8807

Performance Comparison in Monolingual and Cross-Lingual Settings by GPT-3.5 and GPT-4.

Figure1

Performance Comparison Data Visualization.

mance. This observation is also consistent with the resultscross-lingual settings remains a challenge and may demandof COLIEE competitions in recent years, where teams using Japanese as their primary language tend to outperform

more signiﬁcant advancements in model design, access tohigh-quality datasets, and a deeper understanding of thethose using English [1, 2].

complexities involved in cross-lingual tasks.

This ﬁnding aligns with the, earlier mentioned, superior
performance of monolingual settings compared to cross-5Conclusions

lingual ones and highlights the importance of having high-In this study, we evaluated GPT-3.5 and GPT-4 mod-quality translated material and a deep understanding ofels in cross-lingual legal Question-Answering using thethe languages involved when working with cross-lingual

COLIEE Task 4 dataset. Our ﬁndings showed the supe-tasks. It also emphasizes the challenges faced by mod-rior performance of GPT-4 over GPT-3.5 and highlighted

els like GPT-4 and GPT-3.5 in adapting to and dealing

the challenges faced by GPT models in handling linguis-with diﬀerent natural languages and contextual informa-

tic nuances in cross-lingual settings. These insights con-tion, further motivating research and development eﬀorts

tribute to the development of more accurate cross-lingual

to improve their performance for cross-lingual tasks.

QA systems in the legal domain and emphasize the impor-

The experimental results demonstrate the prowess oftance of high-quality translation and deep understanding

the GPT-4 model over its predecessor, GPT-3.5, in bothmonolingual and cross-lingual settings. As advancementsof linguistic complexities. Future research should focus
on improving GPT models’ cross-lingual capabilities andin model architecture and training techniques continue toincorporating domain-speciﬁc knowledge tailored to thebe made, we can expect further performance improve-legal ﬁeld.

ments. However, achieving human-level performance in― 1065 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).

Acknowledgements
This work was supported by the AIP challenge fund-[12] Thi-Hai-Yen Vuong, Hai-Long Nguyen, Tan-MinhNguyen, Hoang-Trung Nguyen, Thai-Binh Nguyen, andHa-Thanh Nguyen. Nowj at coliee 2023–multi-task andensemble approaches in legal information processing. Pro-ing related with JST, AIP Trilateral AI Research, Grant

ceedings of the 10th Competition on Legal Information Extraction/Entailment. COLIEE, 2023.

Number JPMJCR20G4.

[13] Bilgin Onur, Fields Logan, Laverghetta Jr. Antonio, MarjiZaid, Nighojkar Animesh, Steinle Stephen, and Licato
John. Amhr lab 2023 coliee competition approach. Pro-References[1] Juliano Rabelo, Mi-Young Kim, Randy Goebel, Masaharu
Yoshioka, Yoshinobu Kano, and Ken Satoh. A summaryof the COLIEE 2019 competition. In New Frontiers in
Artiﬁcial Intelligence, pp. 34–49, Online, 2020. Springer
International Publishing.
[2] Juliano Rabelo, Randy Goebel, Mi-Young Kim, Yoshinobu
Kano, Masaharu Yoshioka, and Ken Satoh. Overview anddiscussion of the competition on legal information extraction/entailment (coliee) 2021. The Review of Socionetwork Strategies, Vol. 16, No. 1, pp. 111–133, feb 2022.
[3] Yoshinobu Kano. Keyword and snippet based yes/no question answering system for coliee 2015. In Ninth International Workshop on Juris-informatics (JURISIN),2015.
[4] Guilherme Moraes Rosa, Ruan Chaves Rodrigues, Roberto
Lotufo, and Rodrigo Nogueira.
Yes, bm25 is astrong baseline for legal case retrieval. arXiv preprint
arXiv:2105.05686, 2021.
[5] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeﬀrey
Dean. Eﬃcient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013.
[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint
arXiv:1810.04805, 2018.
[7] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke
Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint
arXiv:1907.11692, 2019.
[8] Zhenzhong Lan, Mingda Chen, Sebastian Goodman,Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language
representations. arXiv preprint arXiv:1909.11942, 2019.
[9] HT Nguyen, V Tran, and LM Nguyen. A deep learning
approach for statute law entailment task in coliee-2019.

ceedings of the 10th Competition on Legal Information Extraction/Entailment. COLIEE, 2023.

[14] Guilherme Moraes Rosa, Luiz Bonifacio, Vitor Jeronymo,Hugo Abonizio, Roberto Lotufo, and Rodrigo Nogueira.
Billions of parameters are worth more than in-domain
training data: A case study in the legal case entailment
task. arXiv preprint arXiv:2205.15172, 2022.
[15] Andrew Blair-Stanek, Nils Holzenberger, and Benjamin
Van Durme. Can gpt-3 perform statutory reasoning? arXivpreprint arXiv:2302.06100, 2023.
[16] チェジョンミン, 笠井淳吾, 坂口慶祐. 日本の司法試験を題材とした gpt モデルの評価.
[17] Daniel Martin Katz, Michael James Bommarito, Shang
Gao, and Pablo Arredondo. Gpt-4 passes the bar exam.
Available at SSRN 4389233, 2023.
[18] Ha-Thanh Nguyen, Wachara Fungwacharakorn, and Ken
Satoh. Logilaw dataset towards reinforcement learningfrom logical feedback (rllf). In JURIX 2023: The 36th International Conference on Legal Knowledge and Information Systems, Vol. 379 of Frontiers in Artiﬁcial
Intelligence and Applications, pp. 217–226, 2023.

[19] Ha Thanh Nguyen, Randy Goebel, Francesca Toni, Kostas
Stathis, and Ken Satoh. Lawgiba – combining gpt, knowledge bases, and logic programming in a legal assistancesystem. In JURIX 2023: The 36th International Conference on Legal Knowledge and Information Systems,
Vol. 379 of Frontiers in Artiﬁcial Intelligence and Applications, pp. 371–374, 2023.

Proceedings of the 6th Competition on Legal Information Extraction/Entailment. COLIEE, 2019.

[10] May Myo Zin, Nguyen Ha-Thanh, Satoh Ken, Sugawara
Saku, and Fumihito Nishino. Improving translation of casedescriptions into logical fact formulas using legalcasener.
In Proceedings of the 19th International Conferenceon Artiﬁcial Intelligence and Law - ICAIL 2023, pp.
462–466, 2023.
[11] Phi Manh Kien, Ha-Thanh Nguyen, Ngo Xuan Bach,Vu Tran, Minh Le Nguyen, and Tu Minh Phuong. Answering legal questions by learning neural attentive text
representation. Proceedings of the 28th International
Conference on Computational Linguistics, pp. 988–998, 2020.

― 1066 ―This work is licensed by the author(s) under CC BY 4.0(https://creativecommons.org/licenses/by/4.0/).