LexCLiPR: Cross-Lingual Paragraph Retrieval from Legal Judgments
Authors: Santosh T.Y.S.S, Rohit Upadhya, Matthias Grabmair
Accepted at: acl25

LexCLiPR: Cross-Lingual Paragraph Retrieval from Legal Judgments

Rohit Upadhya*, Santosh T.Y.S.S*
School of Computation, Information, and Technology
Technical University of Munich, Germany

Abstract
Efficient retrieval of pinpointed information from case law is crucial for legal professionals but challenging due to the length and complexity of legal judgments. Existing works mostly often focus on retrieving entire cases rather than precise, paragraph-level information. Moreover, multilingual legal practice necessitates cross-lingual retrieval, most works have been limited to monolingual settings. To address these gaps, we introduce LexCLiPR, a cross-lingual dataset for paragraph-level retrieval from European Court of Human Rights (ECtHR) judgments, leveraging multilingual case law guides and distant supervision to curate our dataset. We evaluate retrieval models in a zero-shot setting, revealing the limitations of pre-trained multilingual models for cross-lingual tasks in low-resource languages and the importance of retrieval based post-training strategies. In fine-tuning settings, we observe that two-tower models excel in cross-lingual retrieval, while siamese architectures are better suited for monolingual tasks. Fine-tuning multilingual models on native language queries improves performance but struggles to generalize to unseen legal concepts, highlighting the need for robust strategies to address topical distribution shifts in the legal queries.1

1 Introduction
Searching for relevant information in case law is a fundamental yet time-consuming task for legal professionals, as the case law serves as a primary source of legal precedent, containing interpretations and applications of statutes that shape future rulings and guide legal reasoning. These judgments are typically dense and lengthy, filled with nuanced language and complex arguments, making it challenging to locate the exact information needed. Research indicates that lawyers spend approximately 15 hours per week reviewing case law, (Lastres, 2015), which accounts for almost 30% of their annual working hours (Poje, 2014), drawing valuable time away from critical tasks. This underscores the need for advanced retrieval systems capable of pinpointing relevant information efficiently so it could significantly reduce research time, allowing legal professionals to focus on deeper analysis, ultimately enhancing their productivity.

Legal information retrieval poses unique challenges that go beyond those of traditional IR due to the complexity and specificity inherent in legal texts. Unlike general documents, legal judgments contain highly structured reasoning, specialized terminology, and intricate argumentation, often tied to jurisdiction-specific doctrines and interpretative frameworks. Moreover, each paragraph’s relevance depend on the contextual understanding of specific legal issues or doctrines, demanding retrievers to go beyond keyword matching to provide meaningful results. Legal queries are typically complex, often requiring systems to interpret layered, implicit legal questions that span multiple legal aspects and contexts. Furthermore, case law is dynamic, with laws evolving and interpretations shifting over time, leading to an ever evolving array of legal concepts, making it essential for retrievers to comprehend new queries and determine relevance.

While much of the research in legal retrieval has focused primarily on retrieving entire cases based on complete-case queries aimed at identifying similar precedents (Santosh et al., 2024b; Ma et al., 2021; Mandal et al., 2017; Goebel et al., 2023; Joshi et al., 2023), there has been a recent shift towards finer-grained retrieval tasks such as paragraph-based retrieval from lengthy legal documents, where either the entire case (Rabelo et al., 2022) or specific, targeted legal queries serve as the query input (Santosh et al., 2024c). This paragraph-

1The dataset and code is available at https://github.com/rohit-upadhya/lexclipr
*These authors contributed equally to this work.
level retrieval addresses the need for retrieving precise information from within a judgment, aligning more closely with the detailed, context-specific information that legal professionals often require. Despite these advances, most studies in legal retrieval have been monolingual, where queries and document corpora share the same language.

As legal practice globalizes, retrieval systems must support diverse linguistic needs, especially in multi-jurisdictional settings. Supra-national courts like the ECHR, CJEU, ICC, and AfCHPR, as well as national courts in multilingual countries like India, serve regions where professionals may submit queries in their native languages, even if judgments are in one of the court’s primary languages. These demands underscore the need for retrieval models that bridge language barriers and map queries to relevant content accurately across languages. Addressing cross-lingual queries requires advanced retrieval systems that navigate both linguistic and legal complexities, ensuring accessible legal knowledge for diverse communities.

To investigate the ability of current retrieval models to identify relevant paragraphs in cross-lingual way, a high-quality labeled dataset is imperative. In this study, we employ distant supervision to construct LexCLiPR, a dataset tailored for query-based relevant paragraph extraction from European Court of Human Rights (ECtHR) judgments, which address alleged violations of rights protected under the European Convention on Human Rights. We leverage the ECtHR’s Knowledge Sharing platform, which provides case law guides in multiple languages, to curate a cross-lingual dataset spanning seven languages. Using the section headers from these guides as queries, we draw on the discussions under each section, which contain paragraph-level citations to English ECtHR judgments, as our relevance signal. Furthermore, we design dataset splits to evaluate the generalizability of retrieval systems on new legal concepts (not seen during training), providing insights into how well these models adapt to the dynamic nature of law.

We evaluate the performance of current multilingual models on our cross-lingual retrieval task in a zero-shot setting. Our findings indicate that: (i) general pre-trained models (e.g., mBERT) and those further pre-trained on legal corpora (e.g., mLegalBERT) underperform compared to models further fine-tuned on general retrieval datasets (e.g., mDPR i.e., mBERT fine-tuned on mMARCO). This highlights the importance of retrieval-specific fine-tuning. (ii) Multilingual models perform better with English-translated queries than with native-language queries, highlighting significant challenges in cross-lingual semantic alignment, particularly for low-resource languages. (iii) Monolingual models such as BERT and DPR consistently outperform multilingual models, even with translated queries, underscoring the trade-off between broader multilingual coverage and language-specific depth in multilingual models.

Further, we fine-tune monolingual models on English-translated data and multilingual models on both original and translated queries using siamese and two-tower architectures. Our key observations include: (i) Two-tower models generally excel in cross-lingual retrieval, while siamese architectures are more effective for monolingual retrieval. (ii) mDPR fine-tuned and tested on native-language queries outperforms its performance on translated queries or monolingual models, suggesting that fine-tuning can mitigate language disparities in multilingual models. (iii) However, this advantage diminishes on an unseen query split, indicating the need for more robust strategies to generalize across unseen topic distributions while maintaining language alignment.

2 Related Work

Legal IR Efficiently retrieving critical legal information is essential for lawyers, spanning tasks such as locating relevant legislation either through specific searches or by providing factual descriptions to identify pertinent statutes (Wang et al., 2018; Paul et al., 2022; Louis and Spanakis, 2021; Santosh et al., 2024d). These tasks extend to retrieving similar past cases (Rabelo et al., 2022; Mandal et al., 2017), civil codes (Kim et al., 2016, 2014), litigation documents for tasks like technology-assisted review (Cormack et al., 2010; Baron et al., 2006), patents (Piroi et al., 2013), and within a firm’s internal support system (Moens, 2001). Our research centers specifically on legal case retrieval. While most existing legal retrieval systems focus on retrieving entire cases (Sansone and Sperlí, 2022) based on various query types—such as whole cases (Rabelo et al., 2022; Ma et al., 2021; Mandal et al., 2017; Joshi et al., 2023; Santosh et al., 2024b) or legal-specific queries (Locke et al., 2017; Locke and Zuccon, 2018; Koniaris et al., 2016)—our approach retrieves relevant paragraphs at a finer level of granularity, allowing practitioners to access tar-
geted information. At the paragraph level, tasks like the legal case entailment task in COLIEE involve finding paragraphs that align with a new case’s decision (Rabelo et al., 2022), using the entire case as a query, unlike the short queries we use, inspired from recent work of Santosh et al. (2024c). This paragraph-level retrieval is critical for building legal question-answering systems (Khazaeli et al., 2021; Sovrano et al., 2021; Verma et al., 2020) and query-focused summarization systems (Santosh et al., 2024a). While prior retrieval studies predominantly focus on monolingual retrieval, where query and documents share the same language, our work advances cross-lingual retrieval for legal documents, addressing multilingual challenges in legal information access.

Cross Lingual IR Cross-Lingual Information Retrieval (CLIR) involves retrieving documents in which search queries and target documents are in different languages (Hull and Grefenstette, 1996). Traditionally, translation-based approaches tackle CLIR by translating either the query or document into a common language, leveraging external machine translation systems or bilingual dictionaries to then use monolingual retrieval methods (McCarley, 1999; Oard, 1998; Zhou et al., 2012; Nair et al., 2020). Recently, neural end-to-end CLIR approaches have emerged, utilizing cross-lingual word embeddings (Vulic and Moens, 2015; Zhang et al., 2019; Litschko et al., 2018). With advances in unsupervised language modeling, models like Multilingual BERT (Devlin, 2018), XLM-R (Conneau, 2019), and Multilingual T5 (Xue, 2020) have been leveraged to extract cross-lingual representations. Transfer learning techniques applied to these cross-lingual embeddings help mitigate the scarcity of non-English data (Van Nguyen et al., 2021; Shi and Lin, 2019; Nair et al., 2020; Schuster et al., 2018). In contrast to the extensive monolingual IR resources, cross-lingual IR datasets are limited. The first CLIR collection emerged with manually translated English queries into German (Salton, 1970). Over time, community-driven evaluations through TREC (Davis and Dunning, 1995; Schäuble and Sheridan, 1998; Voorhees and Harman, 2000), CLEF (Peters, 2019), NCTIR (Kando et al., 1999), and FIRE (Majumder et al., 2010) further enriched CLIR resources. Later, automated pipelines for dataset creation and large multilingual corpora such as Common Crawl enabled the development of datasets like HC4 (Lawrie et al., 2022), HC3 (Lawrie et al., 2023), WikiCLIR (Schamoni et al., 2014), CLIR-Matrix (Sun and Duh, 2020), Large Scale CLIR (Sasaki et al., 2018), and AfriCLIRMatrix (Ogundepo et al., 2022). In this study, we introduce LexCLiPR, a new dataset tailored to advancing CLIR research specifically for legal text collections.

Tasks on ECHR corpora Prior works on the ECtHR corpus have explored diverse tasks, including judgment prediction (Aletras et al., 2016; Chalkidis et al., 2019, 2021; Santosh et al., 2022, 2023a,b, 2024f,e), argument mining (Mochales and Moens, 2008; Mochales and Ieven, 2009; Habernal et al., 2023; Poudyal et al., 2019, 2020; Held and Habernal, 2023), legal reasoning (Chlapanik et al., 2024), event extraction (Filtz et al., 2020; Navas-Loro and Rodriguez-Doncel, 2022), vulnerability classification (Xu et al., 2023), summarization (Santosh et al., 2024a, 2025a), prior case retrieval (Santosh et al., 2024b, 2025b), and relevant paragraph retrieval (Santosh et al., 2024c). While many of these studies focus on judgment documents, recent datasets, like the one by Santosh et al. (2024c) for paragraph retrieval, draw from case law guides curated by the ECtHR registry. Notably, all these works emphasize English. In contrast, we leverage multilingual case law guides to develop our CLIR dataset, contributing a valuable resource to the research community.

3 Dataset

Our task of relevant paragraph extraction from legal judgements is defined as follows: Given a query \( Q \) and a judgement document \( J \) composed of \( n \) paragraphs \( P_J = \{p_1, p_2, \ldots, p_n\} \), the objective is to identify the subset of paragraphs \( P_J^+ \in P_J \) which are relevant to the query.

3.1 Dataset Curation Pipeline

To create LexCLiPR, we leverage case-law guides from the ECtHR Knowledge Sharing Platform*, a resource managed by the Court’s registry that tracks case law evolution across individual convention articles (e.g., Article 9 - Freedom of Thought, Conscience, and Religion) and transversal themes (e.g., Terrorism, Mass Protests, LGBTI Rights). Below, we outline our pipeline for transforming these case law guides into a structured dataset containing query collections and relevant paragraphs within each referenced ECHR judgement. These

*https://www.echr.coe.int/knowledge-sharing
guides, available in multiple languages*, enable the curation of a cross-lingual dataset. Here, the queries derived from these guides are presented in various languages, while the referenced ECHR judgements remain in English.

Judgements Collection We rely on the recent ECHR case collection from Santosh et al. (2024b), sourced from HUDOC*, the ECtHR’s public database. This collection consists of judgments in English, which we segment into individual paragraphs according to the paragraph numbers located at the beginning of each paragraph, providing unique identifiers for cross-referencing. Following Santosh et al. (2024c), we apply hand-crafted heuristics to manage challenges such as inconsistent HTML structures, nested sub-paragraphs, and spurious numbering introduced by verbatim quoting text from other documents to reference them.

Query Collection The case law guides outline key legal concepts under each article or theme, presenting them in a hierarchical structure, with sub-concepts further detailing each concept. A representative index structure of a case law guide is illustrated in Figure 1. For example, this is the hierarchical path of concepts within the Turkish theme guide of Terör (Terrorism) → İstihbarat Aşamasından Eylem Aşamasına Geçiş (Moving on from the surveillance stage to the active phase) → Devlet görevlileri tarafından ölümcül güce başvurulması (Use of lethal force by agents of the State) to … → Devlet görevlilerinin seçimi ve eğitimi (Training and selection of State agents). We extract this table of content hierarchy from the PDF guides. Then we construct query by concatenating these multiple concepts along the path (from the article or theme title to the leaf node in the PDF structure) by using a delimiter, to maintain clarity by providing context. This approach generates structured queries that closely mirror the types of concept lists legal professionals typically search for and use for indexing cases in legal analytics databases.

Relevant Paragraphs in Judgements Each legal concept in the guides is discussed in detail, with references provided to relevant paragraphs within specific ECtHR judgments. An example of this concept description, with cross-references to relevant paragraphs in specific judgments, is illustrated from Turkish guide in App. Fig. 2. We gather all paragraph references within a specific judgment under each concept and label them as relevant to the associated query within that judgment. It is important to note that not all judgments are referenced in these guides, as the focus is primarily on key cases contributing to substantial developments in the law. Thus, for our dataset, we pair each query exclusively with judgments explicitly referenced in the guides and derive their corresponding relevant paragraphs within those specific judgements. While our methodology could, in theory, be applied across all judgments in the corpus, we intentionally limit each query to only the specific judgments mentioned under the legal concept in the case law guides. This approach to query-judgment pairing ensures high-quality relevance, reducing false negatives in the evaluation setup and enhancing the dataset’s reliability.

Finally, we filter the query-judgment pairs to exclude any that lack paragraph-level references. Subsequently, we map each remaining query-judgment pair back to our judgment collection, ensuring that we exclude any references to non-English documents that fall outside our English-only judgment dataset collection.

3.2 Dataset Splits & Analysis

We curate the LexCLiPR dataset across seven languages—English, French, Italian, Romanian, Russian, Turkish, and Ukrainian—utilizing their respective case law guides. This results in a total of 27718 query-judgment pairs, with 7313 unique queries. The distribution of query-judgment pairs across each language is presented in Table 1, showing the highest count in English (7874 query-judgment pairs) and the lowest in Russian (1222 query-judgment pairs). The number of paragraphs in each judgment ranges from 28 to 942, with a mean of 122.46 (Fig. 3a). The percentage of relevant paragraphs in each query-judgment pair varies from 0.11% to 19.64% of the total number of paragraphs in that judgment, with a mean around 2.36%, as depicted in Fig. 4a. The average lengths of queries and paragraphs are 54.19 tokens and 140.04 tokens, respectively, illustrated in Figures 6a and 5a. Detailed language-specific distribution plots can be found in Appendix A.2.

We partition the article/theme case law guides from each language into two distinct splits, ensuring that the first split for each language contains a minimum of 5% of the total query-judgment pairs. This first split consists of query-judgment pairs used exclusively for testing, referred to as "Unseen
<table>
  <tr>
    <th></th>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romm.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td>Unique queries</td>
    <td>1889</td>
    <td>1269</td>
    <td>971</td>
    <td>1334</td>
    <td>207</td>
    <td>1223</td>
    <td>420</td>
  </tr>
  <tr>
    <td>Q-J Pairs</td>
    <td>7874</td>
    <td>4097</td>
    <td>3292</td>
    <td>4313</td>
    <td>1222</td>
    <td>5023</td>
    <td>1897</td>
  </tr>
  <tr>
    <td>Avg #Para. per judgement</td>
    <td>123.24</td>
    <td>124.39</td>
    <td>127.01</td>
    <td>129.46</td>
    <td>111.82</td>
    <td>116.84</td>
    <td>112.76</td>
  </tr>
  <tr>
    <td>Avg. % rel para. per Q-J pair</td>
    <td>2.41</td>
    <td>2.38</td>
    <td>2.46</td>
    <td>2.34</td>
    <td>2.16</td>
    <td>2.25</td>
    <td>2.35</td>
  </tr>
  <tr>
    <td>Unique queries in unseen test</td>
    <td>54</td>
    <td>43</td>
    <td>21</td>
    <td>46</td>
    <td>48</td>
    <td>92</td>
    <td>39</td>
  </tr>
  <tr>
    <td>Unseen test Q-J pairs</td>
    <td>473</td>
    <td>346</td>
    <td>207</td>
    <td>415</td>
    <td>192</td>
    <td>317</td>
    <td>145</td>
  </tr>
  <tr>
    <td>Train Q-J Pairs</td>
    <td>5887</td>
    <td>2965</td>
    <td>2450</td>
    <td>3085</td>
    <td>818</td>
    <td>3746</td>
    <td>1392</td>
  </tr>
  <tr>
    <td>Validation Q-J Pairs</td>
    <td>757</td>
    <td>393</td>
    <td>317</td>
    <td>405</td>
    <td>106</td>
    <td>480</td>
    <td>179</td>
  </tr>
  <tr>
    <td>Seen Test Q-J Pairs</td>
    <td>757</td>
    <td>393</td>
    <td>318</td>
    <td>408</td>
    <td>106</td>
    <td>480</td>
    <td>181</td>
  </tr>
</table>

Table 1: Statistics of LexCLiPR dataset. rel., para. denote relevant and paragraphs respectively.

Legal Queries." It evaluates the model’s performance on unfamiliar legal concepts that it has not encountered during training. The second split referred to as ‘Seen Legal Queries’ is further divided into training, validation, and test sets. This test set assesses the model’s understanding of familiar legal concepts—those seen during training—when applied to new judgments in the test set. To prevent any leakage of unseen test concepts during training, we ensure that none of the case law guides in the unseen split for any language overlap with the seen split of other languages. This guarantees that all unseen concepts remain entirely unfamiliar, even across languages. This approach enforces stricter unfamiliarity, particularly in the context of using translation-based methods for cross-lingual modeling. Details of the case law guides included in the unseen and seen splits across each language are presented in Appendix 4. Statistics on the number of query-judgment pairs and unique queries in the unseen and seen (training, validation, and test) splits are provided in Table 1.

4 Retrieval Methods

We outline the frameworks utilized in our task. Our approach involves calculating a relevance score for each paragraph in the judgment relative to the query, followed by selecting the top-k most relevant paragraphs based on these scores.

Lexical Retrieval We employ BM25 (Robertson et al., 1995), a bag-of-words model that assesses the relevance of paragraphs to queries by analyzing the presence of query terms within the paragraphs. Although this method cannot be directly applied in our cross-lingual setting due to the mismatch between the languages of the queries and documents, it can be applied in combination with a translation model that converts the queries into English, the language of our document corpus.

Dense Retrieval We employ neural bi-encoders to encode queries and paragraphs into low-dimensional representations capturing their semantic content. The final relevance score is computed using the dot product of the representations from the encoders as \( \text{rel}(q, p) = E_q(q).E_p(p) \) where \( E_q \) and \( E_p \) represent query and paragraph encoder respectively. The training objective of retrievers is to learn representations such that relevant pairs of queries and paragraphs exhibit higher similarity than irrelevant ones. To mitigate the computational burden due to the abundance of irrelevant paragraphs, we utilize negative sampling. Let \( \{<q_i; p_i^+, p_{i,1}^-,\ldots,p_{i,n}^->\}_{i=1}^m \) be the training data consisting of m instances with each instance consisting of one query \( q_i \) and one relevant passage \( p_i^+ \), along with n irrelevant (negative) passages \( p_{i,j}^- \). Note these negative paragraphs for a query are sampled from the same document as positive, in our task setup. We optimize the negative log-likelihood loss as follows:

\[
L = -log(\frac{\exp(\text{rel}(q_i, p_i^+))}{\exp(\text{rel}(q, p_i^+)) + \sum_{j=1}^n \exp(\text{rel}(q, p_{i,j}^-)))}
\]

Following Karpukhin et al. 2020, we consider negative samples drawn from randomly selected irrelevant paragraphs to the query, from the same judgment as the positive sample. This approach, Dense Passage Retrieval (DPR) can be implemented in two ways: (i) Siamese (Reimers, 2019; Xiong et al., 2020), which uses a single model to map both the query and document into a shared dense vector space, and (ii) Two-tower (Karpukhin et al., 2020), which employs two independent models to encode the query and document separately into distinct embedding spaces.
5 Experiments

Metrics In accordance with Santosh et al. (2024c), we evaluate performance using Recall@k% (R@K%), which measures the proportion of relevant paragraphs within the top-k% of all paragraphs in a judgment. We report the mean Recall@k% across all instances for \( k = \{2, 5, 10\} \). Utilizing k as a percentage rather than an absolute value accommodates the varying number of paragraphs across different judgments. Higher recall scores indicate better performance.

5.1 Zero-shot

Models We assess zero-shot performance using both multilingual and monolingual models. Our multilingual models include (A, B) mBERT (Devlin, 2018), (E, F) mDPR (Zhang et al., 2021, 2022), which is a multilingual adaptation of DPR (Karpukhin et al., 2020) with BERT replaced by mBERT and further fine-tuned on the English MS MARCO dataset (Bajaj et al., 2016), (H, I) mLegal-BERT (Niklaus et al., 2023) which is continually pre-trained XLM-R model on multilingual legal corpus. All these multilingual models evaluated by using queries in their original languages (A, E, H) as well as English-translated queries (B, F, I) to simulate monolingual retrieval. Our monolingual models include (C) BERT (Devlin, 2018) and (G) DPR (Karpukhin et al., 2020), which is trained on the English Natural Questions dataset (Lee et al., 2019; Kwiatkowski et al., 2019); these models are tested using English-translated queries. We also incorporate a lexical retrieval method, BM25 (D), with English-translated queries for comparison. To handle query translation from the original language to English, we employ the NLLB model (Costajussà et al., 2022), supporting over 200 languages. Note that mDPR follows a shared Siamese architecture, while DPR is a two-tower architecture.

5.1.1 Results

We report Recall@5% results for seen split queries in Table 2, with the unseen split results in Appendix Table 9. Note that for zero-shot experiments, both splits are treated as unseen, as the models are not fine-tuned on any task-specific data. Additional metrics can be found in Appendix Tables 5-10. In a monolingual retrieval setup, models need to generate well-aligned semantic embeddings within a single language to achieve strong performance. In contrast, for cross-lingual retrieval, models must not only produce well-aligned embeddings within each language but also maintain alignment across languages to effectively handle cross-lingual tasks. Key findings are summarized as follows: (i) Multilingual Models with Native Language Queries (A, E, H): Among multilingual models, mDPR (E) outperforms mBERT (A) and mLegal (H) across most languages, highlighting the benefits of retrieval-specific fine-tuning. This advantage is particularly strong in English, where mDPR benefits from continued fine-tuning on the English MSMARCO retrieval dataset. However, mBERT and mLegal perform better in Turkish and Ukrainian, suggesting that mDPR’s English-centric fine-tuning may degrade performance in low-resource languages. These findings emphasize the need for continual training strategies that enhance multilingual capabilities without weakening performance in low-resource languages. (ii) Multilingual Models with Translated Queries (B, F, I): Using English-translated queries improves performance across most languages for multilingual models. While mDPR performs better with English-translated queries (F) than with native ones (E)—which is expected given its fine-tuning on English retrieval corpora—it is notable that mBERT and mLegal (A, H) also perform worse with native queries, despite their multilingual pre-training. This suggests that multilingual models still prioritize English-centric embeddings, likely due to the dominance of English data during pre-training and also highlight significant challenges in cross-lingual semantic alignment, especially for low-resource languages, highlighting the need for improved cross-lingual training objectives to better align multilingual embedding spaces. (iii) Monolingual Models with Translated Queries (C, G): Monolingual models outperform multilingual models even when using translated queries (B, F, I), highlighting the trade-off in multilingual pre-training, which sacrifices depth in individual language representation for broader coverage. Among monolingual models, DPR outperforms BERT, consistent with earlier observations where mBERT and mLegal lag behind mDPR. This reinforces the limitations of standard MLM-based pre-training for retrieval and the need for retrieval-specific fine-tuning strategies. While DPR and BM25 each excel on different query sets (Tab. 2,9), this variability suggests that a hybrid retrieval approach—combining lexical (BM25) and dense (DPR) matching techniques—could effectively handle diverse query types. (iv) Overall,
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train Data</th>
    <th rowspan="2">Model Config</th>
    <th colspan="10">Test Data</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td colspan="12"><b>Zero-shot</b></td>
    <td></td>
  </tr>
  <tr>
    <td>A</td>
    <td>mBERT</td>
    <td>Ori.</td>
    <td>19.91</td>
    <td>17.48</td>
    <td>21.62</td>
    <td>18.79</td>
    <td>16.94</td>
    <td>16.46</td>
    <td>15.17</td>
    <td>17.72</td>
    <td></td>
  </tr>
  <tr>
    <td>B</td>
    <td></td>
    <td>Trans.</td>
    <td>19.91</td>
    <td>19.16</td>
    <td>19.26</td>
    <td>21.24</td>
    <td>19.58</td>
    <td>21.54</td>
    <td>14.96</td>
    <td>19.72</td>
    <td></td>
  </tr>
  <tr>
    <td>C</td>
    <td>BERT</td>
    <td>Trans.</td>
    <td>23.72</td>
    <td>21.65</td>
    <td>23.06</td>
    <td>28.03</td>
    <td>25.18</td>
    <td>25.16</td>
    <td><b>26.41</b></td>
    <td>24.74</td>
    <td></td>
  </tr>
  <tr>
    <td>D</td>
    <td>BM25</td>
    <td>Trans.</td>
    <td>25.04</td>
    <td>23.17</td>
    <td>19.64</td>
    <td>25.32</td>
    <td>21.12</td>
    <td>17.44</td>
    <td>17.29</td>
    <td>21.29</td>
    <td></td>
  </tr>
  <tr>
    <td>E</td>
    <td>Ori.</td>
    <td>29.83</td>
    <td>22.45</td>
    <td>20.68</td>
    <td>22.75</td>
    <td><b>29.79</b></td>
    <td>14.84</td>
    <td>17.57</td>
    <td>22.56</td>
    <td></td>
  </tr>
  <tr>
    <td>F</td>
    <td>Trans.</td>
    <td>29.83</td>
    <td><b>27.55</b></td>
    <td><b>24.70</b></td>
    <td>27.71</td>
    <td>25.22</td>
    <td>23.82</td>
    <td>18.45</td>
    <td><b>25.33</b></td>
    <td></td>
  </tr>
  <tr>
    <td>G</td>
    <td>DPR</td>
    <td>Trans.</td>
    <td><b>28.85</b></td>
    <td><b>25.27</b></td>
    <td><b>23.78</b></td>
    <td><b>28.55</b></td>
    <td><b>29.47</b></td>
    <td><b>26.57</b></td>
    <td><b>22.28</b></td>
    <td><b>26.40</b></td>
    <td></td>
  </tr>
  <tr>
    <td>H</td>
    <td>mLegal</td>
    <td>Ori.</td>
    <td>18.10</td>
    <td>17.16</td>
    <td>17.76</td>
    <td>21.77</td>
    <td>18.39</td>
    <td>20.35</td>
    <td>21.55</td>
    <td>19.30</td>
    <td></td>
  </tr>
  <tr>
    <td>I</td>
    <td></td>
    <td>Trans.</td>
    <td>18.10</td>
    <td>17.98</td>
    <td>18.48</td>
    <td>21.28</td>
    <td>22.15</td>
    <td>19.35</td>
    <td>17.01</td>
    <td>19.19</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="12"><b>Fine-tuning</b></td>
    <td></td>
  </tr>
  <tr>
    <td>a</td>
    <td>Ori.</td>
    <td>Ori.</td>
    <td><b>44.02</b></td>
    <td>41.67</td>
    <td>40.17</td>
    <td>41.17</td>
    <td>36.79</td>
    <td>34.88</td>
    <td>38.44</td>
    <td>39.59</td>
    <td></td>
  </tr>
  <tr>
    <td>b</td>
    <td>Siam.</td>
    <td>Trans.</td>
    <td><b>44.02</b></td>
    <td>42.74</td>
    <td>41.27</td>
    <td><b>48.08</b></td>
    <td>50.14</td>
    <td>42.37</td>
    <td>41.14</td>
    <td>44.25</td>
    <td></td>
  </tr>
  <tr>
    <td>c</td>
    <td>Trans.</td>
    <td>Ori.</td>
    <td>40.74</td>
    <td>41.53</td>
    <td>42.83</td>
    <td>39.19</td>
    <td>41.64</td>
    <td>43.68</td>
    <td>41.10</td>
    <td>41.53</td>
    <td></td>
  </tr>
  <tr>
    <td>d</td>
    <td></td>
    <td>Trans.</td>
    <td>40.74</td>
    <td>44.42</td>
    <td>41.95</td>
    <td>43.41</td>
    <td>53.20</td>
    <td><b>44.39</b></td>
    <td>46.33</td>
    <td>44.92</td>
    <td></td>
  </tr>
  <tr>
    <td>e</td>
    <td>mBERT</td>
    <td>Ori.</td>
    <td>37.59</td>
    <td>39.78</td>
    <td>40.03</td>
    <td>44.99</td>
    <td>51.16</td>
    <td>39.23</td>
    <td>42.06</td>
    <td>42.12</td>
    <td></td>
  </tr>
  <tr>
    <td>f</td>
    <td></td>
    <td>Trans.</td>
    <td>37.59</td>
    <td>40.31</td>
    <td>39.58</td>
    <td>44.75</td>
    <td>49.86</td>
    <td>39.86</td>
    <td>40.77</td>
    <td>41.82</td>
    <td></td>
  </tr>
  <tr>
    <td>g</td>
    <td>Two-tow</td>
    <td>Ori.</td>
    <td>38.22</td>
    <td>39.37</td>
    <td>40.18</td>
    <td>40.48</td>
    <td>50.56</td>
    <td>39.67</td>
    <td>39.78</td>
    <td>41.18</td>
    <td></td>
  </tr>
  <tr>
    <td>h</td>
    <td></td>
    <td>Trans.</td>
    <td>38.22</td>
    <td>40.27</td>
    <td>37.26</td>
    <td>40.75</td>
    <td>50.65</td>
    <td>41.03</td>
    <td>40.47</td>
    <td>41.24</td>
    <td></td>
  </tr>
  <tr>
    <td>i</td>
    <td>BERT</td>
    <td>Trans.</td>
    <td>40.83</td>
    <td>44.25</td>
    <td>41.17</td>
    <td>44.85</td>
    <td><b>55.76</b></td>
    <td>44.11</td>
    <td>44.45</td>
    <td>45.06</td>
    <td></td>
  </tr>
  <tr>
    <td>j</td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>40.46</td>
    <td>43.73</td>
    <td><b>45.30</b></td>
    <td>45.09</td>
    <td>49.26</td>
    <td>44.07</td>
    <td>44.86</td>
    <td>44.68</td>
    <td></td>
  </tr>
  <tr>
    <td>k</td>
    <td>Ori.</td>
    <td>Ori.</td>
    <td>42.15</td>
    <td><b>45.08</b></td>
    <td>44.95</td>
    <td>45.19</td>
    <td>48.93</td>
    <td>44.16</td>
    <td><b>48.91</b></td>
    <td><b>45.62</b></td>
    <td></td>
  </tr>
  <tr>
    <td>m</td>
    <td>Siam.</td>
    <td>Trans.</td>
    <td><b>42.15</b></td>
    <td>44.74</td>
    <td>43.62</td>
    <td>41.63</td>
    <td>50.93</td>
    <td><b>47.48</b></td>
    <td>43.67</td>
    <td>44.89</td>
    <td></td>
  </tr>
  <tr>
    <td>n</td>
    <td>Trans.</td>
    <td>Ori.</td>
    <td>40.63</td>
    <td>39.20</td>
    <td>42.54</td>
    <td>42.64</td>
    <td>29.58</td>
    <td>42.95</td>
    <td>36.45</td>
    <td>39.14</td>
    <td></td>
  </tr>
  <tr>
    <td>o</td>
    <td></td>
    <td>Trans.</td>
    <td>40.63</td>
    <td>41.06</td>
    <td>42.19</td>
    <td>45.54</td>
    <td>48.59</td>
    <td>41.62</td>
    <td>43.79</td>
    <td>43.35</td>
    <td></td>
  </tr>
  <tr>
    <td>p</td>
    <td>mDPR</td>
    <td>Ori.</td>
    <td>40.08</td>
    <td>42.89</td>
    <td>43.18</td>
    <td>46.29</td>
    <td>53.37</td>
    <td>43.26</td>
    <td>41.48</td>
    <td>44.36</td>
    <td></td>
  </tr>
  <tr>
    <td>q</td>
    <td></td>
    <td>Trans.</td>
    <td>40.08</td>
    <td>42.92</td>
    <td>43.74</td>
    <td>46.15</td>
    <td><b>55.11</b></td>
    <td>42.89</td>
    <td>41.40</td>
    <td>44.61</td>
    <td></td>
  </tr>
  <tr>
    <td>r</td>
    <td>Two-tow</td>
    <td>Ori.</td>
    <td>40.56</td>
    <td>40.15</td>
    <td>43.86</td>
    <td>41.46</td>
    <td>43.15</td>
    <td>39.99</td>
    <td>43.07</td>
    <td>41.75</td>
    <td></td>
  </tr>
  <tr>
    <td>s</td>
    <td></td>
    <td>Trans.</td>
    <td>40.56</td>
    <td>40.20</td>
    <td>43.56</td>
    <td>40.97</td>
    <td>44.92</td>
    <td>38.64</td>
    <td>41.45</td>
    <td>41.47</td>
    <td></td>
  </tr>
  <tr>
    <td>t</td>
    <td>DPR</td>
    <td>Trans.</td>
    <td>41.41</td>
    <td>43.53</td>
    <td>43.99</td>
    <td>42.03</td>
    <td>51.96</td>
    <td>41.28</td>
    <td><b>47.68</b></td>
    <td>44.55</td>
    <td></td>
  </tr>
  <tr>
    <td>u</td>
    <td>Ori.</td>
    <td>Ori.</td>
    <td>32.65</td>
    <td>37.63</td>
    <td>40.16</td>
    <td>37.94</td>
    <td>45.62</td>
    <td>36.55</td>
    <td>38.75</td>
    <td>38.47</td>
    <td></td>
  </tr>
  <tr>
    <td>v</td>
    <td>Siam.</td>
    <td>Trans.</td>
    <td>32.65</td>
    <td>34.39</td>
    <td>31.83</td>
    <td>26.82</td>
    <td>31.81</td>
    <td>34.77</td>
    <td>34.80</td>
    <td>32.44</td>
    <td></td>
  </tr>
  <tr>
    <td>w</td>
    <td>Trans.</td>
    <td>Ori.</td>
    <td>37.18</td>
    <td>36.10</td>
    <td>39.46</td>
    <td>40.64</td>
    <td>37.76</td>
    <td>05.91</td>
    <td>34.49</td>
    <td>33.08</td>
    <td></td>
  </tr>
  <tr>
    <td>x</td>
    <td></td>
    <td>Trans.</td>
    <td>37.18</td>
    <td>36.33</td>
    <td>39.38</td>
    <td>38.36</td>
    <td>38.11</td>
    <td>36.52</td>
    <td>35.09</td>
    <td>37.28</td>
    <td></td>
  </tr>
  <tr>
    <td>y</td>
    <td>mLegal</td>
    <td>Ori.</td>
    <td>39.16</td>
    <td>40.49</td>
    <td>43.50</td>
    <td>41.60</td>
    <td>49.86</td>
    <td>39.24</td>
    <td>41.77</td>
    <td>42.23</td>
    <td></td>
  </tr>
  <tr>
    <td>z</td>
    <td></td>
    <td>Trans.</td>
    <td>39.16</td>
    <td>40.13</td>
    <td>43.08</td>
    <td>42.00</td>
    <td>47.66</td>
    <td>40.07</td>
    <td>42.05</td>
    <td>42.02</td>
    <td></td>
  </tr>
  <tr>
    <td>aa</td>
    <td>Two-tow</td>
    <td>Ori.</td>
    <td>37.11</td>
    <td>35.95</td>
    <td>40.35</td>
    <td>38.23</td>
    <td>44.87</td>
    <td>35.97</td>
    <td>39.68</td>
    <td>38.88</td>
    <td></td>
  </tr>
  <tr>
    <td>ab</td>
    <td></td>
    <td>Trans.</td>
    <td>37.11</td>
    <td>37.28</td>
    <td>43.46</td>
    <td>39.56</td>
    <td>44.16</td>
    <td>39.88</td>
    <td>41.71</td>
    <td>40.45</td>
    <td></td>
  </tr>
</table>

Table 2: Recall@5% performance on seen legal queries test split.

monolingual retrieval outperforms cross-lingual retrieval, underscoring the challenges of multilingual representations. Given their complementary strengths, future research should explore ensemble of monolingual and cross-lingual retrieval methods to achieve more robust performance.

5.2 Fine-Tuning

Models We fine-tune three multilingual models—mDPR, mBERT, and mLegal—on our training dataset using queries from all language splits, exploring both Siamese and two-tower architectural framework for each of the model. We experiment
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train Data</th>
    <th rowspan="2">Model Config</th>
    <th colspan="8">Test Data</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td colspan="11"><b>Fine-tuning</b></td>
    <td></td>
  </tr>
  <tr>
    <td>a</td>
    <td>Ori.</td>
    <td>Siam.</td>
    <td>Ori.</td>
    <td>27.68</td>
    <td>27.89</td>
    <td>31.07</td>
    <td>29.61</td>
    <td>34.78</td>
    <td>29.64</td>
    <td>35.63</td>
    <td>30.90</td>
  </tr>
  <tr>
    <td>b</td>
    <td>mBERT</td>
    <td></td>
    <td>Trans.</td>
    <td>27.68</td>
    <td>28.75</td>
    <td>33.16</td>
    <td>30.41</td>
    <td>36.02</td>
    <td>33.39</td>
    <td>37.53</td>
    <td>32.42</td>
  </tr>
  <tr>
    <td>e</td>
    <td>Ori.</td>
    <td>Two-tow</td>
    <td>Ori.</td>
    <td>22.34</td>
    <td>22.85</td>
    <td>25.86</td>
    <td>23.17</td>
    <td>26.15</td>
    <td>25.12</td>
    <td>31.26</td>
    <td>25.25</td>
  </tr>
  <tr>
    <td>f</td>
    <td></td>
    <td></td>
    <td>Trans.</td>
    <td>22.34</td>
    <td>22.83</td>
    <td>26.12</td>
    <td>24.22</td>
    <td>27.43</td>
    <td>24.67</td>
    <td>31.95</td>
    <td>25.65</td>
  </tr>
  <tr>
    <td>i</td>
    <td>BERT</td>
    <td>Trans.</td>
    <td>Siam.</td>
    <td><b>31.01</b></td>
    <td>30.14</td>
    <td><u>34.61</u></td>
    <td><b>34.86</b></td>
    <td><b>38.77</b></td>
    <td>32.07</td>
    <td>37.07</td>
    <td><b>34.08</b></td>
  </tr>
  <tr>
    <td>j</td>
    <td></td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>30.49</td>
    <td><b>32.44</b></td>
    <td><b>34.98</b></td>
    <td><u>32.78</u></td>
    <td>33.90</td>
    <td>32.30</td>
    <td><b>40.63</b></td>
    <td>33.93</td>
  </tr>
  <tr>
    <td>k</td>
    <td>Ori.</td>
    <td>Siam.</td>
    <td>Ori.</td>
    <td>25.83</td>
    <td>28.04</td>
    <td>29.79</td>
    <td>27.46</td>
    <td>33.81</td>
    <td>30.08</td>
    <td><u>38.05</u></td>
    <td>30.44</td>
  </tr>
  <tr>
    <td>m</td>
    <td></td>
    <td></td>
    <td>Trans.</td>
    <td>25.83</td>
    <td><u>31.48</u></td>
    <td>30.17</td>
    <td>26.20</td>
    <td><u>38.64</u></td>
    <td>30.29</td>
    <td>33.16</td>
    <td>30.82</td>
  </tr>
  <tr>
    <td>p</td>
    <td>mDPR</td>
    <td>Ori.</td>
    <td>Two-tow</td>
    <td>28.34</td>
    <td>28.40</td>
    <td>30.99</td>
    <td>27.97</td>
    <td>31.39</td>
    <td><u>33.50</u></td>
    <td>32.99</td>
    <td>30.51</td>
  </tr>
  <tr>
    <td>q</td>
    <td></td>
    <td></td>
    <td>Trans.</td>
    <td>28.34</td>
    <td>29.22</td>
    <td>30.35</td>
    <td>29.44</td>
    <td>31.69</td>
    <td><b>33.56</b></td>
    <td>34.71</td>
    <td>31.04</td>
  </tr>
  <tr>
    <td>t</td>
    <td>DPR</td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>28.15</td>
    <td>29.33</td>
    <td>31.25</td>
    <td>30.38</td>
    <td>34.92</td>
    <td>30.68</td>
    <td>34.54</td>
    <td>31.32</td>
  </tr>
  <tr>
    <td>u</td>
    <td>Ori.</td>
    <td>Siam.</td>
    <td>Ori.</td>
    <td>22.51</td>
    <td>27.79</td>
    <td>30.27</td>
    <td>28.35</td>
    <td>30.00</td>
    <td>22.54</td>
    <td>36.38</td>
    <td>28.26</td>
  </tr>
  <tr>
    <td>v</td>
    <td></td>
    <td></td>
    <td>Trans.</td>
    <td>22.51</td>
    <td>23.39</td>
    <td>20.34</td>
    <td>22.42</td>
    <td>18.11</td>
    <td>22.42</td>
    <td>20.98</td>
    <td>21.45</td>
  </tr>
  <tr>
    <td>y</td>
    <td>mLegal</td>
    <td>Ori.</td>
    <td>Ori.</td>
    <td>28.64</td>
    <td>28.92</td>
    <td>31.72</td>
    <td>31.05</td>
    <td>29.59</td>
    <td>33.43</td>
    <td>32.41</td>
    <td>30.82</td>
  </tr>
  <tr>
    <td>z</td>
    <td></td>
    <td>Two-tow</td>
    <td>Trans.</td>
    <td>28.64</td>
    <td>29.40</td>
    <td>31.95</td>
    <td>31.50</td>
    <td>29.21</td>
    <td>31.67</td>
    <td>32.41</td>
    <td>30.68</td>
  </tr>
</table>

Table 3: Recall@5% performance on unseen legal queries test split.

with two training setups: one using queries in their original languages and another using queries translated into English. After fine-tuning, we evaluate the models on both original-language queries and English-translated queries. Row a in Table 2 and 3 represents mBERT with Siamese architecture trained using queries in their original languages and is then evaluated on queries in the original languages. Row f represents mBERT with Two-Tower architecture, trained using queries in original language and tested on English-translated queries. For monolingual models—BERT and DPR—we fine-tune on the English-translated queries in the training set and also evaluated them on English-translated queries in the test set. For BERT, we experiment with both the siamese and the two-tower, while for DPR, we adhere to its base architecture of two-tower structure. Implementation details are presented in App. B.

5.2.1 Results

We report Recall@5% performance for the seen legal query split in Table 2, with detailed results available in Appendix 5-7. Our key findings are as follows: (i) Multilingual Models Fine-Tuned and Tested with Native Language Queries (a, e, k, p, u, y): Two-tower architectures outperform Siamese models by better aligning semantic embeddings across languages through separate encoders. In contrast, the Siamese setup struggles with cross-lingual alignment due to shared encoding constraints and the dominance of English document data during training. However, mDPR in the Siamese setting performs slightly better than in the two-tower mode, suggesting that retrieval-specific fine-tuning can partially compensate for architectural limitations. Across all models, mDPR remains the strongest performer. (ii) Multilingual Models Fine-Tuned with Native Queries and Tested on English Translated Queries (b, f, m, q, v, z): Models in the two-tower setup (f, q, z) underperform compared to their counterparts evaluated on native language queries (e, p, y). This suggests that fine-tuning on native languages is more effective at overcoming language disparities, particularly for low-resource languages than relying on an English translation-based testing framework. However, in the Siamese setting, performance varies: mBERT performs better on translated queries (a vs. b), while mLegal performs better on native queries (u vs. v).

(iii) Multilingual Models Fine-Tuned and Tested with English-Translated Queries (d, h, o, s, x, ab): The Siamese architecture performs well in this setup, as retrieval is monolingual in nature. Comparing these models to their counterparts fine-tuned and tested on native languages (point (i) above) suggests that fine-tuning on native language data effectively bridges the cross-lingual gap in domain-specific text. This contrasts with zero-shot retrieval, where models perform well with translated queries
than in native language. (iv) Multilingual Models Fine-Tuned on English-Translated Queries and Tested with Native language Queries (c, g, n, r, w, aa): As expected, these models perform best with English queries but also achieve comparable results with native language queries. This suggests that pre-training has provided cross-lingual transfer capabilities, allowing legal-domain knowledge learned in English during fine-tuning to transfer to other languages. This transferability is particularly valuable for low-resource languages, where fine-tuning data is limited. (v) Monolingual Models (i, j, t): Monolingual models remain competitive with multilingual models. Notably, mDPR fine-tuned on native language queries (k) performs on par with or better than DPR (t), highlighting the effectiveness of fine-tuning in bridging domain-specific knowledge across low resource languages.

We report Recall@5% performance on the unseen legal query split in Table 3, with detailed results available in Appendix Tables 8-10. (i) Performance on the unseen split shows significant improvement over zero-shot results indicating that fine-tuning helps models learn transferable features that generalize to new topics. However, it is lower than on the seen split, suggesting the need for better handling of query-side distribution shifts and domain adaptation strategies that require minimal labeled data while avoiding overfitting to seen queries.(ii) mBERT (a, b, e, f) performs better in the Siamese setting, while mLegal (u, v, y, z) excels in the two-tower configuration. mDPR (k, m, p, q) achieves comparable performance across both architectures, suggesting that future work should explore when Siamese or two-tower setups are most effective for generalization. (iii) Training on translated queries improves performance for mBERT and mDPR, but not for mLegal (App. 9), possibly due to overfitting to domain-specific linguistic patterns in native language legal texts. (iv) Among monolingual models, BERT (i, j) slightly outperforms DPR (t) and all the multilingual models. The weaker performance of multilingual models highlight the need for more effective methods to improve the generalization of semantic embeddings, for unseen query topics across languages.

6 Conclusion

We introduced LexCLiPR, a cross-lingual dataset designed for the task of paragraph-level retrieval from case law judgements of European Court of Human Rights (ECtHR) based on a legal query. We curate this dataset using the multilingual case-law guides produced by the court’s registry with paragraph-level citations to ECtHR judgements. Our experiments in a zero-shot setting revealed significant limitations in pre-trained multilingual models, especially for low-resource languages and underscored the importance of retrieval-specific fine-tuning. We further demonstrated in fine-tuning, that two-tower models excel in cross-lingual retrieval, while siamese architectures are more suited for monolingual tasks. Fine-tuning multilingual models on native language queries improved performance but struggled to generalize to unseen legal concepts, highlighting the need for more advanced strategies to handle distribution shifts. We hope that both our dataset and the fine-tuned models will be useful to the research community working in the space of legal information retrieval.

Limitations

Our experiments are conducted on the LexCLiPR dataset, which spans seven languages and focuses on European Court of Human Rights (ECtHR) judgments. While this dataset offers valuable insights for cross-lingual retrieval tasks, its focus on a single jurisdiction restricts the applicability of our findings to other legal systems that may differ significantly in terminology, structure, interpretative frameworks, and linguistic nuances. To develop more universally applicable retrieval models, future work should expand to broader datasets that capture the diversity of global legal systems and multilingual complexities.

Furthermore, our study focuses exclusively on the pre-fetcher stage of retrieval systems, which is responsible for retrieving potentially relevant paragraphs. Consequently, our evaluation prioritizes recall-based metrics, while we leave an exploration of the re-ranking stage—which emphasizes precision-based metrics—for future work. A notable limitation is our treatment of paragraphs as independent units during training, which disregards the inter-paragraph and cross-document context that is often critical in legal texts. While segmenting documents into shorter chunks for retrieval is a common practice in information retrieval, this approach can strip paragraphs of essential contextual information, such as that provided by citations, sequential structures, and cross-references. This challenge is especially pronounced in the legal do-
main, where the interplay between paragraphs and documents is fundamental to accurate interpretation and relevance estimation.

Ethics Statement

LexCLiPR dataset, was curated based on the publicly available sources such as case law guides and HuDOC, the official database of the court and it complies with the ECtHR data policy. These decisions, although not anonymized, include the real names of individuals involved. However, our work does not engage with the data in a way that we consider harmful beyond this availability. We acknowledge the potential for biases inherent in legal data, which may arise from systemic factors or representational imbalances in the dataset. It is crucial to scrutinize these biases thoroughly to ensure that the systems developed promote fairness and do not inadvertently reinforce existing inequalities.

References

Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotiuc-Pietro, and Vasileios Lampos. 2016. Predicting judicial decisions of the european court of human rights: A natural language processing perspective. PeerJ computer science, 2:e93.

Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. Ms marco: A human generated machine reading comprehension dataset. arXiv e-prints, pages arXiv–1611.

Jason R Baron, David D Lewis, and Douglas W Oard. 2006. Trec 2006 legal track overview. In TREC. Citeseer.

Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in english. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317–4323.

Ilias Chalkidis, Manos Fergadiotis, Dimitrios Tsarapatsanis, Nikolaos Aletras, Ion Androutsopoulos, and Prodromos Malakasiotis. 2021. Paragraph-level rationale extraction through regularization: A case study on european court of human rights cases. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 226–241.

Odysseas S Chlapanis, Dimitrios Galanis, and Ion Androutsopoulos. 2024. Lar-echr: A new legal argument reasoning task and dataset for cases of the european court of human rights. arXiv preprint arXiv:2410.13352.

A Conneau. 2019. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116.

Gordon V Cormack, Maura R Grossman, Bruce Hedin, and Douglas W Oard. 2010. Overview of the trec 2010 legal track. In TREC.

Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint arXiv:2207.04672.

Mark W Davis and Ted E Dunning. 1995. A trec evaluation of query translation methods for multi-lingual text retrieval. In Fourth Text Retrieval Conference, volume 483.

Jacob Devlin. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.

Erwin Filtz, María Navas-Loro, Cristiana Santos, Axel Polleres, and Sabrina Kirrane. 2020. Events matter: Extraction of events from court decisions. Legal Knowledge and Information Systems, pages 33–42.

Randy Goebel, Yoshinobu Kano, Mi-Young Kim, Juliano Rabelo, Ken Satoh, and Masaharu Yoshioka. 2023. Summary of the competition on legal information, extraction/entailment (coliee) 2023. In Proceedings of the Nineteenth International Conference on Artificial Intelligence and Law, pages 472–480.

Ivan Habernal, Daniel Faber, Nicola Recchia, Sebastian Bretthauer, Iryna Gurevych, Indra Speicher genannt Döhmann, and Christoph Burchard. 2023. Mining legal arguments in court decisions. Artificial Intelligence and Law, pages 1–38.

Lena Held and Ivan Habernal. 2023. Lacour!: Enabling research on argumentation in hearings of the european court of human rights. arXiv preprint arXiv:2312.05061.

David A Hull and Gregory Grefenstette. 1996. Querying across languages: A dictionary-based approach to multilingual information retrieval. In Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, pages 49–57.

Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 7(3):535–547.

Abhinav Joshi, Akshat Sharma, Sai Kiran Tanikella, and Ashutosh Modi. 2023. U-create: Unsupervised case retrieval using events extraction. arXiv preprint arXiv:2307.05260.

Noriko Kando, Kazuko Kuriyama, Toshihiko Nozue, Koji Eguchi, Hiroyuki Kato, and Souichiro Hidaka.
1999. Overview of ir tasks at the first ntcir workshop. In Proceedings of the first NTCIR workshop on research in Japanese text retrieval and term recognition, pages 11–44.

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781.

Soha Khazaeli, Janardhana Punuru, Chad Morris, Sanjay Sharma, Bert Staub, Michael Cole, Sunny Chiu-Webster, and Dhruv Sakalley. 2021. A free format legal question answering system. In Proceedings of the Natural Legal Language Processing Workshop 2021, pages 107–113.

Mi-Young Kim, Ying Xu, Randy Goebel, and Ken Satoh. 2014. Answering yes/no questions in legal bar exams. In New Frontiers in Artificial Intelligence: JSAI-isAI 2013 Workshops, LENLS, JURISIN, MIMI, AAA, and DDS, Kanagawa, Japan, October 27–28, 2013, Revised Selected Papers 5, pages 199–213. Springer.

Mi-Young Kim, Ying Xu, Yao Lu, and Randy Goebel. 2016. Legal question answering using paraphrasing and entailment analysis. In Tenth International Workshop on Juris-informatics (JURISIN).

Marios Koniaris, Ioannis Anagnostopoulos, and Yannis Vassiliou. 2016. Multi-dimension diversification in legal information retrieval. In Web Information Systems Engineering-WISE 2016: 17th International Conference, Shanghai, China, November 8-10, 2016, Proceedings, Part I 17, pages 174–189. Springer.

Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453–466.

Steven A Lastres. 2015. Rebooting legal research in a digital age.

Dawn Lawrie, James Mayfield, Douglas W Oard, and Eugene Yang. 2022. Hc4: A new suite of test collections for ad hoc clir. In European Conference on Information Retrieval, pages 351–366. Springer.

Dawn Lawrie, James Mayfield, Douglas W Oard, Eugene Yang, Suraj Nair, and Petra Galuščáková. 2023. Hc3: A suite of test collections for clir evaluation over informal text. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2880–2889.

Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. arXiv preprint arXiv:1906.00300.

Robert Litschko, Goran Glavaš, Simone Paolo Ponzetto, and Ivan Vulić. 2018. Unsupervised cross-lingual information retrieval using monolingual data only. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1253–1256.

Daniel Locke and Guido Zuccon. 2018. A test collection for evaluating legal case law search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1261–1264.

Daniel Locke, Guido Zuccon, and Harrisen Scells. 2017. Automatic query generation from legal texts for case law retrieval. In Information Retrieval Technology: 13th Asia Information Retrieval Societies Conference, AIRS 2017, Jeju Island, South Korea, November 22–24, 2017, Proceedings 13, pages 181–193. Springer.

Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. CoRR, abs/1711.05101.

Antoine Louis and Gerasimos Spanakis. 2021. A statutory article retrieval dataset in french. arXiv preprint arXiv:2108.11792.

Yixiao Ma, Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2021. Lecard: a legal case retrieval dataset for chinese law system. In Proceedings of the 44th international ACM SIGIR conference on research and development in information retrieval, pages 2342–2348.

Prasenjit Majumder, Mandar Mitra, Dipasree Pal, Ayan Bandyopadhyay, Samaresh Maiti, Sukomal Pal, Deboshree Modak, and Sucharita Sanyal. 2010. The fire 2008 evaluation exercise. ACM Transactions on Asian Language Information Processing (TALIP), 9(3):1–24.

Arpan Mandal, Kripabandhu Ghosh, Arnab Bhattacharya, Arindam Pal, and Saptarshi Ghosh. 2017. Overview of the fire 2017 ired track: Information retrieval from legal documents. In FIRE (Working Notes), pages 63–68.

J Scott McCarley. 1999. Should we translate the documents or the queries in cross-language information retrieval? In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 208–214.

Raquel Mochales and Aagje Ieven. 2009. Creating an argumentation corpus: do theories apply to real arguments? a case study on the legal argumentation of the echr. In Proceedings of the 12th international conference on artificial intelligence and law, pages 21–30.

Raquel Mochales and Marie-Francine Moens. 2008. Study on the structure of argumentation in case law. In Proceedings of the 2008 conference on legal knowledge and information systems, pages 11–20.
Marie-Francine Moens. 2001. Innovative techniques for legal text retrieval. Artificial Intelligence and Law, 9:29–57.

Suraj Nair, Petra Galuscakova, and Douglas W Oard. 2020. Combining contextualized and non-contextualized query translations to improve clir. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1581–1584.

Maria Navas-Loro and Victor Rodriguez-Doncel. 2022. Whenthefact: Extracting events from european legal decisions. In Legal Knowledge and Information Systems, pages 219–224. IOS Press.

Joel Niklaus, Veton Matoshi, Matthias Stürmer, Ilias Chalkidis, and Daniel E Ho. 2023. Multilegalpile: A 689gb multilingual legal corpus. arXiv preprint arXiv:2306.02069.

Douglas W Oard. 1998. A comparative study of query and document translation for cross-language information retrieval. In Conference of the Association for Machine Translation in the Americas, pages 472–483. Springer.

Odunayo Ogundepo, Xinyu Zhang, Shuo Sun, Kevin Duh, and Jimmy Lin. 2022. Africlimatrix: Enabling cross-lingual information retrieval for african languages. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8721–8728.

Shounak Paul, Pawan Goyal, and Saptarshi Ghosh. 2022. Lesicin: A heterogeneous graph-based approach for automatic legal statute identification from indian legal documents. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 11139–11146.

Carol Peters. 2019. Information retrieval evaluation in a changing world lessons learned from 20 years of clef.

Florina Piroi, Mihai Lupu, and Allan Hanbury. 2013. Overview of clef-ip 2013 lab: Information retrieval in the patent domain. In Information Access Evaluation. Multilinguality, Multimodality, and Visualization: 4th International Conference of the CLEF Initiative. CLEF 2013, Valencia, Spain, September 23-26, 2013. Proceedings 4, pages 232–249. Springer.

Joshua Poje. 2014. Legal research. American Bar Association Techreport, 2014.

Prakash Poudyal, Teresa Gonçalves, and Paulo Quaresma. 2019. Using clustering techniques to identify arguments in legal documents. ASAIL@ ICAIL, 2385.

Prakash Poudyal, Jaromír Šavelka, Aagje Ieven, Marie Francine Moens, Teresa Goncalves, and Paulo Quaresma. 2020. Echr: Legal corpus for argument mining. In Proceedings of the 7th Workshop on Argument Mining, pages 67–75.

Juliano Rabelo, Randy Goebel, Mi-Young Kim, Yoshinobu Kano, Masaharu Yoshioka, and Ken Satoh. 2022. Overview and discussion of the competition on legal information extraction/entailment (coliee) 2021. The Review of Socionetwork Strategies, 16(1):111–133.

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.

Gerard Salton. 1970. Automatic processing of foreign language documents. Journal of the American Society for Information Science, 21(3):187–194.

Carlo Sansone and Giancarlo Sperli. 2022. Legal information retrieval systems: State-of-the-art and open issues. Information Systems, 106:101967.

TYS Santosh, Mahmoud Aly, and Matthias Grabmair. 2024a. Lexabsumm: Aspect-based summarization of legal decisions. arXiv preprint arXiv:2404.00594.

TYS Santosh, Mahmoud Aly, Oana Ichim, and Matthias Grabmair. 2025a. Lexgenie: Automated generation of structured reports for european court of human rights case law. arXiv preprint arXiv:2503.03266.

TYS Santosh, Rashid Gustav Haddad, and Matthias Grabmair. 2024b. Eethr-perc: A dataset for precedent understanding and prior case retrieval in the european court of human rights. arXiv preprint arXiv:2404.00596.

TYS Santosh, Elvin Quero Hernandez, and Matthias Grabmair. 2024c. Query-driven relevant paragraph extraction from legal judgments. arXiv preprint arXiv:2404.00595.

TYS Santosh, Kristina Kaiser, and Matthias Grabmair. 2024d. Cusines: Curriculum-driven structure induced negative sampling for statutory article retrieval. arXiv preprint arXiv:2404.00590.

TYSS Santosh, Irtiza Chowdhury, Shanshan Xu, and Matthias Grabmair. 2024e. The craft of selective prediction: Towards reliable case outcome classification—an empirical study on european court of human rights cases. arXiv preprint arXiv:2409.18645.

TYSS Santosh, Mohamed Hesham Elganayni, Stanislaw Sójka, and Matthias Grabmair. 2024f. Incorporating precedents for legal judgement prediction on european court of human rights cases. arXiv preprint arXiv:2409.18644.

Tyss Santosh, Oana Ichim, and Matthias Grabmair. 2023a. Zero-shot transfer of article-aware legal outcome classification for european court of human rights cases. In Findings of the Association for Computational Linguistics: EACL 2023, pages 593–605.
TYSS Santosh, Isaac Misael OlguÁn Nolasco, and Matthias Grabmair. 2025b. Lecopcr: Legal concept-guided prior case retrieval for european court of human rights cases. arXiv preprint arXiv:2501.14114.

Tyss Santosh, Marcel Perez San Blas, Phillip Kemper, and Matthias Grabmair. 2023b. Leveraging task dependency and contrastive learning for case outcome classification on european court of human rights cases. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1103–1111.

Tyss Santosh, Shanshan Xu, Oana Ichim, and Matthias Grabmair. 2022. Deconfounding legal judgment prediction for european court of human rights cases towards better alignment with experts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 1120–1138.

Shota Sasaki, Shuo Sun, Shigehiko Schamoni, Kevin Duh, and Kentaro Inui. 2018. Cross-lingual learning-to-rank with shared representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 458–463.

Shigehiko Schamoni, Felix Hieber, Artem Sokolov, and Stefan Riezler. 2014. Learning translational and knowledge-based similarities from relevance rankings for cross-language retrieval. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 488–494.

Peter Schäuble and Páraic Sheridan. 1998. Cross-language information retrieval (clir) track overview. NIST SPECIAL PUBLICATION SP, pages 31–44.

Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2018. Cross-lingual transfer learning for multilingual task oriented dialog. arXiv preprint arXiv:1810.13327.

Peng Shi and Jimmy Lin. 2019. Cross-lingual relevance transfer for document retrieval. arXiv preprint arXiv:1911.02989.

Francesco Sovrano, Monica Palmirani, Biagio Distefano, Salvatore Sapienza, and Fabio Vitali. 2021. A dataset for evaluating legal question answering on private international law. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pages 230–234.

Shuo Sun and Kevin Duh. 2020. Clirmatrix: A massively large collection of bilingual and multilingual datasets for cross-lingual information retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4160–4170.

Minh Van Nguyen, Tuan Ngo Nguyen, Bonan Min, and Thien Huu Nguyen. 2021. Crosslingual transfer learning for relation and event extraction via word category and class alignments. In Proceedings of the 2021 conference on empirical methods in natural language processing, pages 5414–5426.

Aayushi Verma, Jorge Morato, Arti Jain, and Anuja Arora. 2020. Relevant subsection retrieval for law domain question answer system. Data Visualization and Knowledge Engineering: Spotting Data Points with Artificial Intelligence, pages 299–319.

Ellen M Voorhees and Donna Harman. 2000. Overview of the sixth text retrieval conference (trec-6). Information Processing & Management, 36(1):3–35.

Ivan Vulic and Marie-Francine Moens. 2015. Bilingual word embeddings from non-parallel document-aligned data applied to bilingual lexicon induction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL 2015), volume 2, pages 719–725. ACL; East Stroudsburg, PA.

Pengfei Wang, Ze Yang, Shuzi Niu, Yongfeng Zhang, Lei Zhang, and ShaoZhang Niu. 2018. Modeling dynamic pairwise attention for crime classification over legal articles. In the 41st international ACM SIGIR conference on research & development in information retrieval, pages 485–494.

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808.

Shanshan Xu, Leon Stauffer, TYSS Santosh, Oana Ichim, Corina Heri, and Matthias Grabmair. 2023. Vechr: A dataset for explainable and robust classification of vulnerability type in the european court of human rights. arXiv preprint arXiv:2310.11368.

L. Xue. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.

Rui Zhang, Caitlin Westerfield, Sungrok Shim, Garrett Bingham, Alexander Fabbri, Neha Verma, William Hu, and Dragomir Radev. 2019. Improving low-resource cross-lingual document retrieval by reranking with deep bilingual representations. arXiv preprint arXiv:1906.03492.

Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. 2021. Mr. tydi: A multi-lingual benchmark for dense retrieval. arXiv preprint arXiv:2108.08787.

Xinyu Zhang, Nandan Thakur, Odunayo Ogundepo, Ehsan Kamalloo, David Alfonso-Hermelo, Xiaoguang Li, Qun Liu, Mehdi Rezagholizadeh, and Jimmy Lin. 2022. Making a miracl: Multilingual information retrieval across a continuum of languages. arXiv preprint arXiv:2210.09984.

Dong Zhou, Mark Truran, Tim Brailsford, Vincent Wade, and Helen Ashman. 2012. Translation techniques in cross-language information retrieval. ACM Computing Surveys (CSUR), 45(1):1–44.
Figure 1: Illustration of the Table of Contents from the Turkish case law guide on Terrorism, facilitating the derivation of legal concept queries for the LexCLiPR dataset.

A Dataset

A.1 Dataset Curation

Figure 1 presents a Table of Contents from the Turkish case law guide on Terrorism. This conceptual mapping serves as a foundation for formulating legal concept queries for the LexCLiPR dataset. Figure 2 highlights the structure of case law guides, where each concept is accompanied by explicit references to relevant paragraphs from ECtHR judgments. These references provide relevance signals of paragraphs in judgments that correspond to these legal concepts.

A.2 Dataset Distribution

Figures 3, 4, 5 and 6 present total dataset (all) and language specific distribution of total number of paragraphs per judgement, total number of relevant paragraphs per judgement, number of tokens per query and number of tokens per paragraph respectively.

A.3 Case law Guides and Data split Breakdown

Table 4 presents a detailed breakdown of ECtHR case law guides in various languages, with their usage across different dataset splits. S, U, - indicate that that the case law guide is used in the ‘Seen getirilmemiş durumunda, oze konusu iddiaların esasına ilişkin somutlara varabileceğini eklemiştir.

2. Devlet görevlileri tarafından önlencel gic kullanılmaması ilişkin yükümlülükler
24. Sözleşme'nin 2. maddesi gereğince, önlencel gic kullanılmaması “mutlaka gerekli” olduğu ölçüde önlencel gic su amaçlarla başvurulması mümkündür; kişinin yaşa dayan bir ihale karşı savunulması, usulline uygun bir yakalama işlemi yapılması veya usulle uygulan olası bir ihale karışımsal bir sonuç elde etmesi, ya da bir yolsuzluk veya bir haksızlık veya uygulam şeklide başlanmış Mahkeme yetkililerinin, Sözleşme'nin 2. maddesinde belirtilen “mutlaka gerekli” ifadesinin, Sözleşme'nin 8-11. maddeleri anlamında “demokratik bir toplumda gerekli” ifadesinden daha bir gerçeklik kriteri olduğu vurgulanmaktadır
(Ölcümsel-önerge/Bitirme Kararları, § 149).

Figure 2: Illustration of Contents of a case law guide, illustrating how legal concepts are discussed with explicit references to relevant paragraphs in ECtHR judgments, enabling the derivation of relevance signals for LexCLiPR.

Legal Queries’ data split, ‘Unseen Legal Queries’ data split and unavailable respectively.

B Implementation Details

For our zero-shot baseline, we utilize BM25 with hyperparameters \( k1 = 1.5 \) and \( b = 0.75 \). For dense models, we employ max pooling to aggregate the hidden state representations of all tokens from the final layer and use cosine similarity as the similarity function. We use FAISS vector datastore (Johnson et al., 2019) for efficient retrieval. The models are fine-tuned with 7 randomly sampled negatives per query. We sweep through learning rates within the range of {1e−6, 5e−6, 1e−5, 5e−5} for fine-tuning. The model is trained for 5 epochs using the AdamW optimizer (Loshchilov and Hutter, 2017), and the best model is selected based on the validation results. We use the same setup to train all these dense models: BERT*, mBERT*, DPR* *, mDPR*, mLegalBERT* models.

*www.huggingface.co/google-bert/bert-base-uncased
*www.huggingface.co/google-bert/bert-base-multilingual-cased
*www.huggingface.co/facebook/dpr-ctx_encoder-single-nq-base
*www.huggingface.co/facebook/dpr-question_encoder-single-nq-base
*www.huggingface.co/castorini/mdpr-tied-pft-msmarco
*www.huggingface.co/joelniiklaus/legal-xlm-roberta-base
<table>
  <tr>
    <th>ECHR Guide Title</th>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr><td>Social Rights</td><td>S</td><td>S</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Article 6 (Criminal Limb)</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td></tr>
  <tr><td>Rights of LGBTI Persons</td><td>S</td><td>S</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Article 1 of Protocol No. 1</td><td>S</td><td>S</td><td>-</td><td>S</td><td>S</td><td>S</td><td>S</td></tr>
  <tr><td>Article 13</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Prisoners’ Rights</td><td>S</td><td>-</td><td>-</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 2 of Protocol No. 4</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Article 11</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 18</td><td>U</td><td>U</td><td>U</td><td>U</td><td>-</td><td>U</td><td>-</td></tr>
  <tr><td>Article 4</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>S</td></tr>
  <tr><td>Article 4 of Protocol No. 7</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td></tr>
  <tr><td>Article 5</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td></tr>
  <tr><td>Article 17</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 15</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td></tr>
  <tr><td>Article 14 and Article 1 of Protocol No. 12</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 12</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Immigration</td><td>U</td><td>U</td><td>-</td><td>U</td><td>-</td><td>U</td><td>-</td></tr>
  <tr><td>Article 34/35</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Environment</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Mass Protests</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 3 of Protocol No. 1</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>S</td></tr>
  <tr><td>Article 8</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 6 (Civil Limb)</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>S</td></tr>
  <tr><td>Article 3</td><td>U</td><td>U</td><td>U</td><td>U</td><td>U</td><td>U</td><td>U</td></tr>
  <tr><td>Article 1</td><td>S</td><td>S</td><td>S</td><td>-</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 10</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 7</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td><td>S</td></tr>
  <tr><td>Article 9</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Terrorism</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td></tr>
  <tr><td>Article 2 of Protocol No. 7</td><td>S</td><td>-</td><td>-</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Article 46</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Article 3 of Protocol No. 4</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Article 1 of Protocol No. 7</td><td>S</td><td>-</td><td>-</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Article 2</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>S</td></tr>
  <tr><td>Data Protection</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Article 2 of Protocol No. 1</td><td>S</td><td>S</td><td>S</td><td>S</td><td>-</td><td>S</td><td>S</td></tr>
  <tr><td>Article 4 of Protocol No. 4</td><td>S</td><td>S</td><td>-</td><td>S</td><td>-</td><td>S</td><td>S</td></tr>
  <tr><td>Article 1 of Protocol No. 7</td><td>S</td><td>-</td><td>-</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
  <tr><td>Rights of LGBTI Persons</td><td>S</td><td>-</td><td>-</td><td>S</td><td>-</td><td>-</td><td>-</td></tr>
</table>

Table 4: List of ECHR Case law guides with their usage across dataset splits and languages. -, S, U represent unavailable, used in the seen split, and used in the unseen split respectively.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train Data</th>
    <th rowspan="2">Model Config</th>
    <th colspan="7">Test Data</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td colspan="10" style="text-align:center;">Zero-shot</td>
    <td></td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>11.53</td>
    <td>10.14</td>
    <td>10.18</td>
    <td>11.06</td>
    <td>09.62</td>
    <td>08.50</td>
    <td>09.80</td>
    <td>10.12</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>11.53</td>
    <td>11.44</td>
    <td>07.86</td>
    <td>11.61</td>
    <td>10.61</td>
    <td>09.46</td>
    <td>07.82</td>
    <td>10.05</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>Trans.</td>
    <td></td>
    <td>12.83</td>
    <td>12.48</td>
    <td>10.74</td>
    <td>14.78</td>
    <td>13.41</td>
    <td>14.64</td>
    <td>10.84</td>
    <td>12.82</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>Trans.</td>
    <td></td>
    <td>12.24</td>
    <td>12.20</td>
    <td>07.98</td>
    <td>12.19</td>
    <td>08.66</td>
    <td>06.83</td>
    <td>10.32</td>
    <td>10.06</td>
  </tr>
  <tr>
    <td>mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>16.07</td>
    <td>10.69</td>
    <td>09.02</td>
    <td>14.62</td>
    <td>16.95</td>
    <td>06.93</td>
    <td>12.46</td>
    <td>12.39</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>16.07</td>
    <td>14.03</td>
    <td>10.72</td>
    <td>14.67</td>
    <td>18.57</td>
    <td>11.29</td>
    <td>09.45</td>
    <td>13.54</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td></td>
    <td><b>17.35</b></td>
    <td><b>14.45</b></td>
    <td><b>14.14</b></td>
    <td><b>15.26</b></td>
    <td><b>17.53</b></td>
    <td><b>15.96</b></td>
    <td><b>13.59</b></td>
    <td><b>15.47</b></td>
  </tr>
  <tr>
    <td>mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>11.51</td>
    <td>08.82</td>
    <td>12.29</td>
    <td>10.24</td>
    <td>11.23</td>
    <td>09.64</td>
    <td>11.83</td>
    <td>10.79</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>11.51</td>
    <td>09.38</td>
    <td>10.49</td>
    <td>09.43</td>
    <td>12.90</td>
    <td>09.94</td>
    <td>09.33</td>
    <td>10.43</td>
  </tr>
  <tr>
    <td colspan="10" style="text-align:center;">Fine-tuning</td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td><b>23.73</b></td>
    <td>23.49</td>
    <td>22.73</td>
    <td>20.09</td>
    <td>18.76</td>
    <td>17.11</td>
    <td>17.75</td>
    <td>20.52</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>23.73</b></td>
    <td>22.60</td>
    <td>22.48</td>
    <td>24.31</td>
    <td>27.04</td>
    <td>23.36</td>
    <td>20.15</td>
    <td>23.38</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>21.70</td>
    <td><b>24.63</b></td>
    <td>24.55</td>
    <td>23.26</td>
    <td>22.51</td>
    <td><b>25.08</b></td>
    <td>23.43</td>
    <td>23.59</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>21.70</td>
    <td>24.11</td>
    <td>21.58</td>
    <td><b>26.12</b></td>
    <td>28.56</td>
    <td>23.80</td>
    <td>25.94</td>
    <td>24.54</td>
  </tr>
  <tr>
    <td rowspan="2">mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>16.73</td>
    <td>17.79</td>
    <td>18.99</td>
    <td>22.19</td>
    <td>20.45</td>
    <td>18.62</td>
    <td>20.56</td>
    <td>19.33</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>16.73</td>
    <td>17.28</td>
    <td>21.04</td>
    <td>20.61</td>
    <td>20.62</td>
    <td>18.48</td>
    <td>21.12</td>
    <td>19.41</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>19.06</td>
    <td>18.64</td>
    <td>19.21</td>
    <td>21.28</td>
    <td>22.38</td>
    <td>19.84</td>
    <td>20.60</td>
    <td>20.14</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>19.06</td>
    <td>20.01</td>
    <td>18.96</td>
    <td>22.24</td>
    <td>23.99</td>
    <td>20.76</td>
    <td>21.65</td>
    <td>20.95</td>
  </tr>
  <tr>
    <td rowspan="2">BERT</td>
    <td>Trans.</td>
    <td>Siam.</td>
    <td><b>26.00</b></td>
    <td>23.38</td>
    <td>23.82</td>
    <td><b>25.69</b></td>
    <td>27.04</td>
    <td><b>25.60</b></td>
    <td>25.66</td>
    <td><b>25.31</b></td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>22.20</td>
    <td>21.97</td>
    <td>23.67</td>
    <td>25.57</td>
    <td><b>32.55</b></td>
    <td>22.59</td>
    <td>26.15</td>
    <td>24.96</td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td>23.15</td>
    <td>21.69</td>
    <td><b>26.40</b></td>
    <td>23.65</td>
    <td>25.55</td>
    <td>23.14</td>
    <td>23.62</td>
    <td>23.89</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>23.15</td>
    <td><b>24.73</b></td>
    <td>26.03</td>
    <td>22.58</td>
    <td>27.94</td>
    <td>24.88</td>
    <td>21.18</td>
    <td>24.36</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>22.05</td>
    <td>20.33</td>
    <td>22.84</td>
    <td>25.13</td>
    <td>17.51</td>
    <td>23.95</td>
    <td>21.63</td>
    <td>21.92</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>22.05</td>
    <td>22.71</td>
    <td>21.70</td>
    <td>24.53</td>
    <td><b>28.89</b></td>
    <td>22.78</td>
    <td>24.82</td>
    <td>23.93</td>
  </tr>
  <tr>
    <td rowspan="2">mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>20.12</td>
    <td>20.79</td>
    <td>19.02</td>
    <td>24.32</td>
    <td>26.04</td>
    <td>22.15</td>
    <td>21.75</td>
    <td>22.03</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>20.12</td>
    <td>21.03</td>
    <td>20.23</td>
    <td>23.93</td>
    <td>27.77</td>
    <td>21.81</td>
    <td>23.43</td>
    <td>22.62</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>19.96</td>
    <td>21.44</td>
    <td>20.71</td>
    <td>22.28</td>
    <td>22.30</td>
    <td>19.55</td>
    <td>18.82</td>
    <td>20.72</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>19.96</td>
    <td>18.83</td>
    <td>21.84</td>
    <td>19.65</td>
    <td>23.78</td>
    <td>19.58</td>
    <td>20.87</td>
    <td>20.64</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>20.51</td>
    <td>22.63</td>
    <td>23.20</td>
    <td>21.67</td>
    <td>25.83</td>
    <td>20.57</td>
    <td><b>29.75</b></td>
    <td>23.45</td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td>15.26</td>
    <td>16.86</td>
    <td>18.66</td>
    <td>19.78</td>
    <td>21.91</td>
    <td>15.99</td>
    <td>18.88</td>
    <td>18.19</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>15.26</td>
    <td>15.91</td>
    <td>14.93</td>
    <td>13.25</td>
    <td>16.87</td>
    <td>17.33</td>
    <td>15.72</td>
    <td>15.61</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>17.66</td>
    <td>17.91</td>
    <td>19.61</td>
    <td>20.65</td>
    <td>20.78</td>
    <td>2.37</td>
    <td>20.52</td>
    <td>17.07</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>17.66</td>
    <td>19.62</td>
    <td>19.36</td>
    <td>18.26</td>
    <td>17.89</td>
    <td>20.99</td>
    <td>17.66</td>
    <td>18.78</td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>18.76</td>
    <td>19.21</td>
    <td>19.85</td>
    <td>19.58</td>
    <td>18.06</td>
    <td>21.39</td>
    <td>18.23</td>
    <td>19.30</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>18.76</td>
    <td>18.93</td>
    <td>20.85</td>
    <td>20.67</td>
    <td>17.94</td>
    <td>20.79</td>
    <td>18.05</td>
    <td>19.43</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>16.13</td>
    <td>17.86</td>
    <td>21.58</td>
    <td>18.52</td>
    <td>19.80</td>
    <td>18.38</td>
    <td>19.99</td>
    <td>18.89</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>16.13</td>
    <td>18.99</td>
    <td>24.00</td>
    <td>19.58</td>
    <td>26.53</td>
    <td>19.43</td>
    <td>24.02</td>
    <td>21.24</td>
  </tr>
</table>

Table 5: Recall@2% performance on seen legal queries test split.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train Data</th>
    <th rowspan="2">Model Config</th>
    <th colspan="6">Test Data</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td colspan="10"><b>Zero-shot</b></td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>19.91</td>
    <td>17.48</td>
    <td>21.62</td>
    <td>18.79</td>
    <td>16.94</td>
    <td>16.46</td>
    <td>15.17</td>
    <td>17.72</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>19.91</td>
    <td>19.16</td>
    <td>19.26</td>
    <td>21.24</td>
    <td>19.58</td>
    <td>21.54</td>
    <td>14.96</td>
    <td>19.72</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>Trans.</td>
    <td></td>
    <td>23.72</td>
    <td>21.65</td>
    <td>23.06</td>
    <td><b>28.03</b></td>
    <td>25.18</td>
    <td><b>25.16</b></td>
    <td><b>26.41</b></td>
    <td>24.74</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>Trans.</td>
    <td></td>
    <td>25.04</td>
    <td>23.17</td>
    <td>19.64</td>
    <td>25.32</td>
    <td>21.12</td>
    <td>17.44</td>
    <td>17.29</td>
    <td>21.29</td>
  </tr>
  <tr>
    <td>mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td><b>29.83</b></td>
    <td>22.45</td>
    <td>20.68</td>
    <td>22.75</td>
    <td><b>29.79</b></td>
    <td>14.84</td>
    <td>17.57</td>
    <td>22.56</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td><b>29.83</b></td>
    <td><b>27.55</b></td>
    <td><b>24.70</b></td>
    <td>27.71</td>
    <td>25.22</td>
    <td>23.82</td>
    <td>18.45</td>
    <td><b>25.33</b></td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td></td>
    <td><b>28.85</b></td>
    <td><b>25.27</b></td>
    <td><b>23.78</b></td>
    <td><b>28.55</b></td>
    <td><b>29.47</b></td>
    <td><b>26.57</b></td>
    <td><b>22.28</b></td>
    <td><b>26.40</b></td>
  </tr>
  <tr>
    <td>mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>18.10</td>
    <td>17.16</td>
    <td>17.76</td>
    <td>21.77</td>
    <td>18.39</td>
    <td>20.35</td>
    <td>21.55</td>
    <td>19.30</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>18.10</td>
    <td>17.98</td>
    <td>18.48</td>
    <td>21.28</td>
    <td>22.15</td>
    <td>19.35</td>
    <td>17.01</td>
    <td>19.19</td>
  </tr>
  <tr>
    <td colspan="10"><b>Fine-tuning</b></td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td><b>44.02</b></td>
    <td>41.67</td>
    <td>40.17</td>
    <td>41.17</td>
    <td>36.79</td>
    <td>34.88</td>
    <td>38.44</td>
    <td>39.59</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>44.02</b></td>
    <td>42.74</td>
    <td>41.27</td>
    <td><b>48.08</b></td>
    <td>50.14</td>
    <td>42.37</td>
    <td>41.14</td>
    <td>44.25</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>40.74</td>
    <td>41.53</td>
    <td>42.83</td>
    <td>39.19</td>
    <td>41.64</td>
    <td>43.68</td>
    <td>41.10</td>
    <td>41.53</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>40.74</td>
    <td>44.42</td>
    <td>41.95</td>
    <td>43.41</td>
    <td>53.20</td>
    <td>44.39</td>
    <td>46.33</td>
    <td>44.92</td>
  </tr>
  <tr>
    <td rowspan="2">mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>37.59</td>
    <td>39.78</td>
    <td>40.03</td>
    <td>44.99</td>
    <td>51.16</td>
    <td>39.23</td>
    <td>42.06</td>
    <td>42.12</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>37.59</td>
    <td>40.31</td>
    <td>39.58</td>
    <td>44.75</td>
    <td>49.86</td>
    <td>39.86</td>
    <td>40.77</td>
    <td>41.82</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>38.22</td>
    <td>39.37</td>
    <td>40.18</td>
    <td>40.48</td>
    <td>50.56</td>
    <td>39.67</td>
    <td>39.78</td>
    <td>41.18</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>38.22</td>
    <td>40.27</td>
    <td>37.26</td>
    <td>40.75</td>
    <td>50.65</td>
    <td>41.03</td>
    <td>40.47</td>
    <td>41.24</td>
  </tr>
  <tr>
    <td rowspan="2">BERT</td>
    <td>Trans.</td>
    <td>Siam.</td>
    <td>40.83</td>
    <td>44.25</td>
    <td>41.17</td>
    <td>44.85</td>
    <td><b>55.76</b></td>
    <td>44.11</td>
    <td>44.45</td>
    <td><b>45.06</b></td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>40.46</td>
    <td>43.73</td>
    <td><b>45.30</b></td>
    <td>45.09</td>
    <td>49.26</td>
    <td>44.07</td>
    <td>44.86</td>
    <td>44.68</td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td><b>42.15</b></td>
    <td><b>45.08</b></td>
    <td>44.95</td>
    <td>45.19</td>
    <td>48.93</td>
    <td>44.16</td>
    <td><b>48.91</b></td>
    <td><b>45.62</b></td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>42.15</b></td>
    <td><b>44.74</b></td>
    <td>43.62</td>
    <td>41.63</td>
    <td>50.93</td>
    <td><b>47.48</b></td>
    <td>43.67</td>
    <td>44.89</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>40.63</td>
    <td>39.20</td>
    <td>42.54</td>
    <td>42.64</td>
    <td>29.58</td>
    <td>42.95</td>
    <td>36.45</td>
    <td>39.14</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>40.63</td>
    <td>41.06</td>
    <td>42.19</td>
    <td>45.54</td>
    <td>48.59</td>
    <td>41.62</td>
    <td>43.79</td>
    <td>43.35</td>
  </tr>
  <tr>
    <td rowspan="2">mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>40.08</td>
    <td>42.89</td>
    <td>43.18</td>
    <td><b>46.29</b></td>
    <td>53.37</td>
    <td>43.26</td>
    <td>41.48</td>
    <td>44.36</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>40.08</td>
    <td>42.92</td>
    <td>43.74</td>
    <td>46.15</td>
    <td><b>55.11</b></td>
    <td>42.89</td>
    <td>41.40</td>
    <td>44.61</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>40.56</td>
    <td>40.15</td>
    <td>43.86</td>
    <td>41.46</td>
    <td>43.15</td>
    <td>39.99</td>
    <td>43.07</td>
    <td>41.75</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>40.56</td>
    <td>40.20</td>
    <td>43.56</td>
    <td>40.97</td>
    <td>44.92</td>
    <td>38.64</td>
    <td>41.45</td>
    <td>41.47</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>41.41</td>
    <td>43.53</td>
    <td>43.99</td>
    <td>42.03</td>
    <td>51.96</td>
    <td>41.28</td>
    <td><b>47.68</b></td>
    <td>44.55</td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td>32.65</td>
    <td>37.63</td>
    <td>40.16</td>
    <td>37.94</td>
    <td><b>45.62</b></td>
    <td>36.55</td>
    <td>38.75</td>
    <td>38.47</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>32.65</td>
    <td>34.39</td>
    <td>31.83</td>
    <td>26.82</td>
    <td>31.81</td>
    <td>34.77</td>
    <td>34.80</td>
    <td>32.44</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>37.18</td>
    <td>36.10</td>
    <td>39.46</td>
    <td>40.64</td>
    <td>37.76</td>
    <td>0.591</td>
    <td>34.49</td>
    <td>33.08</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>37.18</td>
    <td>36.33</td>
    <td>39.38</td>
    <td>38.36</td>
    <td>38.11</td>
    <td>36.52</td>
    <td>35.09</td>
    <td>37.28</td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>39.16</td>
    <td>40.49</td>
    <td>43.50</td>
    <td>41.60</td>
    <td>49.86</td>
    <td>39.24</td>
    <td>41.77</td>
    <td>42.23</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>39.16</td>
    <td>40.13</td>
    <td>43.08</td>
    <td>42.00</td>
    <td>47.66</td>
    <td>40.07</td>
    <td>42.05</td>
    <td>42.02</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>37.11</td>
    <td>35.95</td>
    <td>40.35</td>
    <td>38.23</td>
    <td>44.87</td>
    <td>35.97</td>
    <td>39.68</td>
    <td>38.88</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>37.11</td>
    <td>37.28</td>
    <td>43.46</td>
    <td>39.56</td>
    <td>44.16</td>
    <td>39.88</td>
    <td>41.71</td>
    <td>40.45</td>
  </tr>
</table>

Table 6: Recall@5% performance on seen legal queries test split.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train Data</th>
    <th rowspan="2">Model Config</th>
    <th colspan="7">Test Data</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td colspan="10"><b>Zero-shot</b></td>
    <td></td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>31.55</td>
    <td>29.20</td>
    <td>30.27</td>
    <td>32.47</td>
    <td>30.45</td>
    <td>27.84</td>
    <td>34.22</td>
    <td>30.86</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>31.55</td>
    <td>31.06</td>
    <td>34.70</td>
    <td>32.74</td>
    <td>32.60</td>
    <td>33.52</td>
    <td>28.70</td>
    <td>32.12</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>Trans.</td>
    <td></td>
    <td>35.88</td>
    <td>32.89</td>
    <td>35.18</td>
    <td>39.33</td>
    <td>36.96</td>
    <td><b>38.89</b></td>
    <td><b>41.84</b></td>
    <td>37.28</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>Trans.</td>
    <td></td>
    <td><b>38.89</b></td>
    <td>36.98</td>
    <td>34.74</td>
    <td>38.47</td>
    <td>38.75</td>
    <td>31.64</td>
    <td>29.43</td>
    <td>35.56</td>
  </tr>
  <tr>
    <td>mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td><b>41.81</b></td>
    <td>36.77</td>
    <td>32.09</td>
    <td>36.21</td>
    <td>37.69</td>
    <td>25.61</td>
    <td>27.76</td>
    <td>33.99</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td><b>41.81</b></td>
    <td><b>37.47</b></td>
    <td><b>37.14</b></td>
    <td><b>41.88</b></td>
    <td>39.61</td>
    <td>38.61</td>
    <td>30.87</td>
    <td>38.20</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td></td>
    <td>40.29</td>
    <td><b>38.27</b></td>
    <td><b>35.90</b></td>
    <td><b>44.95</b></td>
    <td><b>43.47</b></td>
    <td><b>39.77</b></td>
    <td><b>34.92</b></td>
    <td><b>39.65</b></td>
  </tr>
  <tr>
    <td>mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>30.62</td>
    <td>29.12</td>
    <td>31.35</td>
    <td>31.79</td>
    <td>30.25</td>
    <td>35.03</td>
    <td>32.29</td>
    <td>31.49</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>30.62</td>
    <td>30.27</td>
    <td>30.99</td>
    <td>32.38</td>
    <td>30.97</td>
    <td>31.19</td>
    <td>31.90</td>
    <td>31.19</td>
  </tr>
  <tr>
    <td colspan="10"><b>Fine-tuning</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>63.45</td>
    <td>65.12</td>
    <td>65.67</td>
    <td>62.91</td>
    <td>58.03</td>
    <td>59.99</td>
    <td>63.07</td>
    <td>62.61</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>63.45</td>
    <td>68.22</td>
    <td>67.24</td>
    <td><b>69.30</b></td>
    <td>75.10</td>
    <td>68.37</td>
    <td><b>69.29</b></td>
    <td><b>68.71</b></td>
  </tr>
  <tr>
    <td rowspan="2">mBERT</td>
    <td>Siam.</td>
    <td></td>
    <td><b>64.80</b></td>
    <td>67.69</td>
    <td>66.49</td>
    <td>63.20</td>
    <td>63.16</td>
    <td>67.34</td>
    <td>61.67</td>
    <td>64.91</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>64.80</b></td>
    <td>66.76</td>
    <td>66.91</td>
    <td>66.71</td>
    <td>77.55</td>
    <td>66.94</td>
    <td>69.08</td>
    <td>68.39</td>
  </tr>
  <tr>
    <td rowspan="2">mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>59.86</td>
    <td>66.40</td>
    <td>66.55</td>
    <td>65.95</td>
    <td>70.73</td>
    <td>63.40</td>
    <td>62.74</td>
    <td>65.09</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>59.86</td>
    <td>65.98</td>
    <td>67.32</td>
    <td>65.93</td>
    <td>72.61</td>
    <td>62.81</td>
    <td>61.81</td>
    <td>65.19</td>
  </tr>
  <tr>
    <td rowspan="2">mBERT</td>
    <td>Two-tow</td>
    <td></td>
    <td>59.08</td>
    <td>64.64</td>
    <td>64.32</td>
    <td>64.73</td>
    <td>71.17</td>
    <td>63.25</td>
    <td>63.12</td>
    <td>64.33</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>59.08</td>
    <td>65.76</td>
    <td>63.93</td>
    <td>64.89</td>
    <td>72.08</td>
    <td>64.47</td>
    <td>63.89</td>
    <td>64.87</td>
  </tr>
  <tr>
    <td rowspan="2">BERT</td>
    <td>Trans.</td>
    <td>Siam.</td>
    <td>60.81</td>
    <td>65.41</td>
    <td>59.64</td>
    <td>62.94</td>
    <td>68.76</td>
    <td>62.99</td>
    <td>65.79</td>
    <td>63.76</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>63.72</td>
    <td><b>68.52</b></td>
    <td>67.65</td>
    <td><b>70.03</b></td>
    <td>77.38</td>
    <td>66.62</td>
    <td>68.93</td>
    <td><b>68.98</b></td>
  </tr>
  <tr>
    <td rowspan="2">mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>64.02</td>
    <td>68.43</td>
    <td>67.60</td>
    <td>67.87</td>
    <td>75.98</td>
    <td>67.77</td>
    <td>66.46</td>
    <td>68.30</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>64.02</td>
    <td>67.56</td>
    <td>65.81</td>
    <td>59.60</td>
    <td>76.42</td>
    <td><b>69.69</b></td>
    <td>63.31</td>
    <td>66.63</td>
  </tr>
  <tr>
    <td rowspan="2">mDPR</td>
    <td>Siam.</td>
    <td></td>
    <td>63.47</td>
    <td>63.97</td>
    <td>66.84</td>
    <td>63.29</td>
    <td>41.84</td>
    <td>69.11</td>
    <td>57.75</td>
    <td>60.90</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>63.47</td>
    <td>63.53</td>
    <td>65.16</td>
    <td>65.82</td>
    <td>72.69</td>
    <td>66.48</td>
    <td>64.85</td>
    <td>66.00</td>
  </tr>
  <tr>
    <td rowspan="2">mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>62.70</td>
    <td>65.31</td>
    <td>67.15</td>
    <td>68.18</td>
    <td>75.78</td>
    <td>65.52</td>
    <td>66.18</td>
    <td>67.26</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>62.70</td>
    <td>65.73</td>
    <td>69.04</td>
    <td>67.72</td>
    <td>76.07</td>
    <td>65.41</td>
    <td>65.65</td>
    <td>67.47</td>
  </tr>
  <tr>
    <td rowspan="2">mDPR</td>
    <td>Two-tow</td>
    <td></td>
    <td>63.47</td>
    <td>66.78</td>
    <td><b>70.79</b></td>
    <td>66.54</td>
    <td>73.20</td>
    <td>66.34</td>
    <td>66.29</td>
    <td>67.63</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>63.47</td>
    <td>66.53</td>
    <td><b>70.41</b></td>
    <td>66.41</td>
    <td>73.61</td>
    <td>66.19</td>
    <td>64.96</td>
    <td>67.37</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>62.91</td>
    <td>66.16</td>
    <td>69.25</td>
    <td>63.59</td>
    <td><b>78.98</b></td>
    <td><b>69.17</b></td>
    <td>68.63</td>
    <td>68.38</td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>56.09</td>
    <td>62.74</td>
    <td>64.56</td>
    <td>63.80</td>
    <td>73.35</td>
    <td>65.69</td>
    <td>63.37</td>
    <td>64.23</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>56.09</td>
    <td>60.17</td>
    <td>54.93</td>
    <td>46.10</td>
    <td>60.38</td>
    <td>60.02</td>
    <td>59.64</td>
    <td>56.76</td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Siam.</td>
    <td></td>
    <td>59.93</td>
    <td>61.37</td>
    <td>59.32</td>
    <td>63.45</td>
    <td>68.95</td>
    <td>12.84</td>
    <td>54.54</td>
    <td>54.34</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>59.93</td>
    <td>61.97</td>
    <td>60.47</td>
    <td>61.43</td>
    <td>68.98</td>
    <td>59.85</td>
    <td>62.96</td>
    <td>62.23</td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>64.25</td>
    <td>66.81</td>
    <td>69.04</td>
    <td>65.30</td>
    <td>70.38</td>
    <td>63.15</td>
    <td>62.51</td>
    <td>65.92</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>64.25</b></td>
    <td>66.38</td>
    <td>69.15</td>
    <td>65.26</td>
    <td>70.38</td>
    <td>62.97</td>
    <td>61.41</td>
    <td>65.69</td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Two-tow</td>
    <td></td>
    <td>62.39</td>
    <td>63.24</td>
    <td>66.32</td>
    <td>64.62</td>
    <td>74.59</td>
    <td>62.79</td>
    <td>66.10</td>
    <td>65.72</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>62.39</td>
    <td>62.71</td>
    <td>66.55</td>
    <td>64.23</td>
    <td>72.00</td>
    <td>66.41</td>
    <td>65.28</td>
    <td>65.65</td>
  </tr>
</table>

Table 7: Recall@10% performance on seen legal queries test split.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train Data</th>
    <th rowspan="2">Model Config</th>
    <th colspan="7">Test Data</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td colspan="10"><b>Zero-shot</b></td>
    <td></td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>08.08</td>
    <td>06.72</td>
    <td>06.72</td>
    <td>07.12</td>
    <td>11.66</td>
    <td>06.46</td>
    <td>09.14</td>
    <td>07.99</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>08.08</td>
    <td>08.35</td>
    <td>10.44</td>
    <td>09.69</td>
    <td>14.40</td>
    <td>13.15</td>
    <td>11.90</td>
    <td>10.86</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>Trans.</td>
    <td></td>
    <td>11.84</td>
    <td>10.76</td>
    <td><b>12.13</b></td>
    <td><b>12.92</b></td>
    <td>10.74</td>
    <td>14.56</td>
    <td>10.52</td>
    <td>11.92</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>Trans.</td>
    <td></td>
    <td><b>14.41</b></td>
    <td>10.03</td>
    <td>07.21</td>
    <td>12.76</td>
    <td>16.52</td>
    <td><b>15.88</b></td>
    <td>11.03</td>
    <td>12.55</td>
  </tr>
  <tr>
    <td>mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>13.91</td>
    <td>09.86</td>
    <td>06.63</td>
    <td>09.99</td>
    <td>15.59</td>
    <td>07.33</td>
    <td>08.51</td>
    <td>10.26</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td><b>13.91</b></td>
    <td><b>14.00</b></td>
    <td><b>12.51</b></td>
    <td><b>13.02</b></td>
    <td><b>18.18</b></td>
    <td><b>16.85</b></td>
    <td>08.62</td>
    <td><b>13.87</b></td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td></td>
    <td>10.69</td>
    <td><b>10.81</b></td>
    <td>11.96</td>
    <td>10.67</td>
    <td><b>20.18</b></td>
    <td>12.22</td>
    <td><b>14.89</b></td>
    <td><b>13.06</b></td>
  </tr>
  <tr>
    <td>mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>09.65</td>
    <td>07.86</td>
    <td>07.75</td>
    <td>11.57</td>
    <td>12.55</td>
    <td>10.40</td>
    <td><b>17.47</b></td>
    <td>11.04</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>09.65</td>
    <td>08.80</td>
    <td>08.48</td>
    <td>10.53</td>
    <td>08.90</td>
    <td>11.27</td>
    <td>07.93</td>
    <td>09.37</td>
  </tr>
  <tr>
    <td colspan="10"><b>Fine-tuning</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="4">mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>13.70</td>
    <td>13.44</td>
    <td>15.81</td>
    <td>14.89</td>
    <td>18.73</td>
    <td>14.99</td>
    <td>18.10</td>
    <td>15.67</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>13.70</td>
    <td>13.28</td>
    <td>15.51</td>
    <td>15.97</td>
    <td>17.83</td>
    <td>16.69</td>
    <td>19.20</td>
    <td>16.03</td>
  </tr>
  <tr>
    <td>Siam.</td>
    <td></td>
    <td><b>15.85</b></td>
    <td>13.57</td>
    <td>13.63</td>
    <td>14.42</td>
    <td>12.42</td>
    <td>14.33</td>
    <td>15.52</td>
    <td>14.25</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>15.85</b></td>
    <td>16.66</td>
    <td>15.76</td>
    <td><b>18.82</b></td>
    <td>14.91</td>
    <td>14.11</td>
    <td><b>21.21</b></td>
    <td>16.76</td>
  </tr>
  <tr>
    <td rowspan="2">BERT</td>
    <td>Ori.</td>
    <td></td>
    <td>10.28</td>
    <td>10.40</td>
    <td>13.35</td>
    <td>10.74</td>
    <td>14.36</td>
    <td>11.14</td>
    <td>16.44</td>
    <td>12.39</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>10.28</td>
    <td>10.50</td>
    <td>12.77</td>
    <td>10.12</td>
    <td>11.08</td>
    <td>10.76</td>
    <td>13.33</td>
    <td>11.26</td>
  </tr>
  <tr>
    <td></td>
    <td>Two-tow</td>
    <td></td>
    <td>13.69</td>
    <td>13.84</td>
    <td>14.62</td>
    <td>14.55</td>
    <td>12.48</td>
    <td>14.68</td>
    <td>16.09</td>
    <td>14.28</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>13.69</td>
    <td>14.06</td>
    <td>15.09</td>
    <td>13.59</td>
    <td>12.51</td>
    <td>14.35</td>
    <td>15.40</td>
    <td>14.10</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td>Siam.</td>
    <td>15.64</td>
    <td>14.50</td>
    <td>16.97</td>
    <td><b>17.82</b></td>
    <td>18.55</td>
    <td>17.77</td>
    <td>16.15</td>
    <td><b>16.77</b></td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>12.44</td>
    <td>13.05</td>
    <td>14.44</td>
    <td>14.01</td>
    <td>15.07</td>
    <td>13.81</td>
    <td>17.93</td>
    <td>14.39</td>
  </tr>
  <tr>
    <td rowspan="4">mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>12.37</td>
    <td>13.47</td>
    <td>17.27</td>
    <td>13.80</td>
    <td>19.98</td>
    <td>15.37</td>
    <td><b>22.82</b></td>
    <td>16.44</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>12.37</td>
    <td>15.07</td>
    <td>14.88</td>
    <td>14.49</td>
    <td><b>20.32</b></td>
    <td>16.72</td>
    <td>20.17</td>
    <td>16.29</td>
  </tr>
  <tr>
    <td>Siam.</td>
    <td></td>
    <td><b>15.92</b></td>
    <td>16.77</td>
    <td>16.50</td>
    <td>15.13</td>
    <td>15.63</td>
    <td>15.20</td>
    <td>16.55</td>
    <td>15.96</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>15.92</b></td>
    <td><b>17.99</b></td>
    <td>17.36</td>
    <td>14.71</td>
    <td>19.09</td>
    <td><b>19.59</b></td>
    <td>17.64</td>
    <td><b>17.47</b></td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>13.88</td>
    <td>12.10</td>
    <td>11.75</td>
    <td>13.02</td>
    <td>12.77</td>
    <td>15.80</td>
    <td>08.51</td>
    <td>12.55</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>13.88</td>
    <td>12.01</td>
    <td>13.01</td>
    <td>14.20</td>
    <td>13.66</td>
    <td>16.79</td>
    <td>11.26</td>
    <td>13.54</td>
  </tr>
  <tr>
    <td></td>
    <td>Two-tow</td>
    <td></td>
    <td>13.22</td>
    <td>13.72</td>
    <td>14.15</td>
    <td>12.86</td>
    <td>15.73</td>
    <td>14.37</td>
    <td>16.55</td>
    <td>14.37</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>13.22</td>
    <td>12.07</td>
    <td>14.24</td>
    <td>14.45</td>
    <td>14.78</td>
    <td>14.84</td>
    <td>19.89</td>
    <td>14.78</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>Trans.</td>
    <td>13.74</td>
    <td>12.96</td>
    <td>14.44</td>
    <td>14.80</td>
    <td>16.35</td>
    <td>12.85</td>
    <td>15.69</td>
    <td>14.40</td>
  </tr>
  <tr>
    <td rowspan="4">mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>09.68</td>
    <td>14.30</td>
    <td>13.47</td>
    <td>16.18</td>
    <td>13.59</td>
    <td>09.71</td>
    <td>16.26</td>
    <td>13.31</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>09.68</td>
    <td>11.11</td>
    <td>08.26</td>
    <td>10.48</td>
    <td>06.72</td>
    <td>09.24</td>
    <td>09.37</td>
    <td>09.27</td>
  </tr>
  <tr>
    <td>Siam.</td>
    <td></td>
    <td>13.06</td>
    <td>13.15</td>
    <td>13.13</td>
    <td>13.73</td>
    <td>14.45</td>
    <td>04.58</td>
    <td>15.63</td>
    <td>12.53</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>13.06</td>
    <td>13.94</td>
    <td>14.03</td>
    <td>15.70</td>
    <td>10.50</td>
    <td>13.13</td>
    <td>13.45</td>
    <td>13.40</td>
  </tr>
  <tr>
    <td></td>
    <td>Ori.</td>
    <td></td>
    <td>13.83</td>
    <td>15.48</td>
    <td><b>19.29</b></td>
    <td>15.07</td>
    <td>14.99</td>
    <td>17.02</td>
    <td>17.53</td>
    <td>16.17</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>13.83</td>
    <td>15.60</td>
    <td><b>19.23</b></td>
    <td>16.07</td>
    <td>14.46</td>
    <td>16.93</td>
    <td>17.53</td>
    <td>16.24</td>
  </tr>
  <tr>
    <td></td>
    <td>Two-tow</td>
    <td></td>
    <td>09.22</td>
    <td>09.76</td>
    <td>12.38</td>
    <td>08.68</td>
    <td>15.15</td>
    <td>10.95</td>
    <td>12.30</td>
    <td>11.21</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>09.22</td>
    <td>09.87</td>
    <td>11.63</td>
    <td>10.19</td>
    <td>15.93</td>
    <td>10.73</td>
    <td>11.15</td>
    <td>11.25</td>
  </tr>
</table>

Table 8: Recall@2% performance on unseen legal queries test split.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train Data</th>
    <th rowspan="2">Model Config</th>
    <th colspan="8">Test Data</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td colspan="10"><b>Zero-shot</b></td>
    <td colspan="3"></td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>16.76</td>
    <td>15.72</td>
    <td>14.33</td>
    <td>14.49</td>
    <td>21.65</td>
    <td>12.65</td>
    <td>15.57</td>
    <td>15.88</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>16.76</td>
    <td>20.02</td>
    <td>20.11</td>
    <td>17.42</td>
    <td>24.49</td>
    <td>20.86</td>
    <td>23.85</td>
    <td>20.50</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>Trans.</td>
    <td></td>
    <td>19.87</td>
    <td>18.78</td>
    <td>21.21</td>
    <td>21.60</td>
    <td>22.70</td>
    <td>25.98</td>
    <td>17.76</td>
    <td>21.13</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>Trans.</td>
    <td></td>
    <td><b>25.09</b></td>
    <td><b>23.12</b></td>
    <td>20.33</td>
    <td><b>27.18</b></td>
    <td><b>36.08</b></td>
    <td><b>26.44</b></td>
    <td><b>27.13</b></td>
    <td><b>26.48</b></td>
  </tr>
  <tr>
    <td>mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>23.59</td>
    <td>17.62</td>
    <td>15.07</td>
    <td>15.68</td>
    <td>26.25</td>
    <td>12.53</td>
    <td>15.06</td>
    <td>17.97</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>23.59</td>
    <td><b>25.23</b></td>
    <td><b>22.29</b></td>
    <td>20.72</td>
    <td>29.91</td>
    <td>25.74</td>
    <td>19.37</td>
    <td>23.84</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td></td>
    <td>20.93</td>
    <td>19.18</td>
    <td>21.78</td>
    <td>20.27</td>
    <td><b>33.11</b></td>
    <td>22.57</td>
    <td>20.40</td>
    <td>22.61</td>
  </tr>
  <tr>
    <td>mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>16.35</td>
    <td>15.19</td>
    <td>16.18</td>
    <td>17.47</td>
    <td>21.86</td>
    <td>20.06</td>
    <td>22.30</td>
    <td>18.49</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>16.35</td>
    <td>16.20</td>
    <td>16.85</td>
    <td>17.07</td>
    <td>19.31</td>
    <td>19.70</td>
    <td>19.31</td>
    <td>17.83</td>
  </tr>
  <tr>
    <td colspan="10"><b>Fine-tuning</b></td>
    <td colspan="3"></td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td>27.68</td>
    <td>27.89</td>
    <td>31.07</td>
    <td>29.61</td>
    <td>34.78</td>
    <td>29.64</td>
    <td>35.63</td>
    <td>30.90</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>27.68</td>
    <td>28.75</td>
    <td>33.16</td>
    <td>30.41</td>
    <td>36.02</td>
    <td>33.39</td>
    <td>37.53</td>
    <td>32.42</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>29.53</td>
    <td>27.40</td>
    <td>29.61</td>
    <td>32.29</td>
    <td>29.31</td>
    <td>31.86</td>
    <td>39.25</td>
    <td>31.32</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>29.53</td>
    <td>28.36</td>
    <td>28.25</td>
    <td>33.70</td>
    <td>29.37</td>
    <td>30.64</td>
    <td><b>41.26</b></td>
    <td>31.59</td>
  </tr>
  <tr>
    <td rowspan="2">mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>22.34</td>
    <td>22.85</td>
    <td>25.86</td>
    <td>23.17</td>
    <td>26.15</td>
    <td>25.12</td>
    <td>31.26</td>
    <td>25.25</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>22.34</td>
    <td>22.83</td>
    <td>26.12</td>
    <td>24.22</td>
    <td>27.43</td>
    <td>24.67</td>
    <td>31.95</td>
    <td>25.65</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>27.09</td>
    <td>28.36</td>
    <td>32.12</td>
    <td>28.42</td>
    <td>33.07</td>
    <td>29.38</td>
    <td>35.11</td>
    <td>30.51</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>27.09</td>
    <td>28.96</td>
    <td>32.15</td>
    <td>29.20</td>
    <td>32.67</td>
    <td>29.54</td>
    <td>35.29</td>
    <td>30.70</td>
  </tr>
  <tr>
    <td rowspan="2">BERT</td>
    <td>Trans.</td>
    <td>Siam.</td>
    <td><b>31.01</b></td>
    <td>30.14</td>
    <td><b>34.61</b></td>
    <td><b>34.86</b></td>
    <td><b>38.77</b></td>
    <td>32.07</td>
    <td>37.07</td>
    <td><b>34.08</b></td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>30.49</td>
    <td>32.44</td>
    <td><b>34.98</b></td>
    <td>32.78</td>
    <td>33.90</td>
    <td>32.30</td>
    <td>40.63</td>
    <td>33.93</td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td>25.83</td>
    <td>28.04</td>
    <td>29.79</td>
    <td>27.46</td>
    <td>33.81</td>
    <td>30.08</td>
    <td>38.05</td>
    <td>30.44</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>25.83</td>
    <td>31.48</td>
    <td>30.17</td>
    <td>26.20</td>
    <td>38.64</td>
    <td>30.29</td>
    <td>33.16</td>
    <td>30.82</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td><b>31.28</b></td>
    <td><b>33.05</b></td>
    <td>33.81</td>
    <td>30.68</td>
    <td>32.84</td>
    <td>34.23</td>
    <td>31.55</td>
    <td>32.49</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>31.28</b></td>
    <td><b>32.96</b></td>
    <td>34.53</td>
    <td>33.04</td>
    <td>38.05</td>
    <td><b>36.66</b></td>
    <td>34.31</td>
    <td><b>34.40</b></td>
  </tr>
  <tr>
    <td rowspan="2">mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>28.34</td>
    <td>28.40</td>
    <td>30.99</td>
    <td>27.97</td>
    <td>31.39</td>
    <td>33.50</td>
    <td>32.99</td>
    <td>30.51</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>28.34</td>
    <td>29.22</td>
    <td>30.35</td>
    <td>29.44</td>
    <td>31.69</td>
    <td>33.56</td>
    <td>34.71</td>
    <td>31.04</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>29.49</td>
    <td>28.68</td>
    <td>29.68</td>
    <td>29.81</td>
    <td>31.85</td>
    <td>31.06</td>
    <td>35.17</td>
    <td>30.82</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>29.49</td>
    <td>29.03</td>
    <td>31.75</td>
    <td>31.48</td>
    <td>33.33</td>
    <td>30.02</td>
    <td>36.21</td>
    <td>31.62</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>28.15</td>
    <td>29.33</td>
    <td>31.25</td>
    <td>30.38</td>
    <td>34.92</td>
    <td>30.68</td>
    <td>34.54</td>
    <td>31.32</td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td>22.51</td>
    <td>27.79</td>
    <td>30.27</td>
    <td>28.35</td>
    <td>30.00</td>
    <td>22.54</td>
    <td>36.38</td>
    <td>28.26</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>22.51</td>
    <td>23.39</td>
    <td>20.34</td>
    <td>22.42</td>
    <td>18.11</td>
    <td>22.42</td>
    <td>20.98</td>
    <td>21.45</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>23.73</td>
    <td>22.34</td>
    <td>29.84</td>
    <td>25.34</td>
    <td>29.03</td>
    <td>11.90</td>
    <td>30.98</td>
    <td>24.74</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>23.73</td>
    <td>25.29</td>
    <td>23.95</td>
    <td>26.31</td>
    <td>24.27</td>
    <td>28.67</td>
    <td>25.00</td>
    <td>25.32</td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>28.64</td>
    <td>28.92</td>
    <td>31.72</td>
    <td>31.05</td>
    <td>29.59</td>
    <td>33.43</td>
    <td>32.41</td>
    <td>30.82</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>28.64</td>
    <td>29.40</td>
    <td>31.95</td>
    <td>31.50</td>
    <td>29.21</td>
    <td>31.67</td>
    <td>32.41</td>
    <td>30.68</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>24.71</td>
    <td>26.27</td>
    <td>27.59</td>
    <td>24.93</td>
    <td>32.65</td>
    <td>24.77</td>
    <td>23.22</td>
    <td>26.31</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>24.71</td>
    <td>25.81</td>
    <td>28.78</td>
    <td>24.94</td>
    <td>31.92</td>
    <td>26.43</td>
    <td>31.09</td>
    <td>27.67</td>
  </tr>
</table>

Table 9: Recall@5% performance on unseen legal queries test split.
<table>
  <tr>
    <th rowspan="2">Model</th>
    <th rowspan="2">Train Data</th>
    <th rowspan="2">Model Config</th>
    <th colspan="7">Test Data</th>
    <th rowspan="2">Avg.</th>
  </tr>
  <tr>
    <th>Eng.</th>
    <th>Fre.</th>
    <th>Ita.</th>
    <th>Romn.</th>
    <th>Rus.</th>
    <th>Turk.</th>
    <th>Ukr.</th>
  </tr>
  <tr>
    <td colspan="10"><b>Zero-shot</b></td>
    <td></td>
  </tr>
  <tr>
    <td>mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>27.93</td>
    <td>25.88</td>
    <td>25.43</td>
    <td>26.91</td>
    <td>31.41</td>
    <td>21.77</td>
    <td>26.55</td>
    <td>26.55</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>27.93</td>
    <td>30.47</td>
    <td>34.25</td>
    <td>29.75</td>
    <td>33.11</td>
    <td>32.2</td>
    <td><b>36.26</b></td>
    <td>32.00</td>
  </tr>
  <tr>
    <td>BERT</td>
    <td>Trans.</td>
    <td></td>
    <td>31.98</td>
    <td>32.07</td>
    <td>35.06</td>
    <td>33.5</td>
    <td>35.93</td>
    <td><b>42.09</b></td>
    <td>32.82</td>
    <td>34.78</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>Trans.</td>
    <td></td>
    <td><b>35.15</b></td>
    <td><b>34.88</b></td>
    <td>31.31</td>
    <td><b>38.2</b></td>
    <td><b>48.52</b></td>
    <td><b>39.67</b></td>
    <td><b>45.29</b></td>
    <td><b>39.00</b></td>
  </tr>
  <tr>
    <td>mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td><b>35.84</b></td>
    <td>30.59</td>
    <td>23.47</td>
    <td>28.55</td>
    <td>39.42</td>
    <td>19.19</td>
    <td>29.54</td>
    <td>29.51</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td><b>35.84</b></td>
    <td><b>37.78</b></td>
    <td><b>36.1</b></td>
    <td>34.91</td>
    <td><b>45.17</b></td>
    <td>38.22</td>
    <td>34.94</td>
    <td><b>37.57</b></td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td></td>
    <td>30.76</td>
    <td>32.31</td>
    <td><b>35.1</b></td>
    <td>31.15</td>
    <td>42.99</td>
    <td>34.96</td>
    <td>33.85</td>
    <td>34.45</td>
  </tr>
  <tr>
    <td>mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>24.37</td>
    <td>24.56</td>
    <td>26.89</td>
    <td>26.82</td>
    <td>30.8</td>
    <td>30.61</td>
    <td>34.6</td>
    <td>28.38</td>
  </tr>
  <tr>
    <td></td>
    <td>Trans.</td>
    <td></td>
    <td>24.37</td>
    <td>24.98</td>
    <td>27.62</td>
    <td>26.29</td>
    <td>27.41</td>
    <td>30.09</td>
    <td>29.37</td>
    <td>27.16</td>
  </tr>
  <tr>
    <td colspan="10"><b>Fine-tuning</b></td>
    <td></td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td><b>48.03</b></td>
    <td>47.15</td>
    <td>47.92</td>
    <td>50.21</td>
    <td>53.83</td>
    <td>46.98</td>
    <td>51.32</td>
    <td>49.35</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>48.03</b></td>
    <td>48.05</td>
    <td>48.08</td>
    <td>48.22</td>
    <td>53.7</td>
    <td>52.6</td>
    <td>55.8</td>
    <td>50.64</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>47.72</td>
    <td>46.98</td>
    <td>49.7</td>
    <td>51.71</td>
    <td>49.09</td>
    <td>52.1</td>
    <td>56.72</td>
    <td>50.57</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>47.72</td>
    <td>48.5</td>
    <td>48.58</td>
    <td><b>54.24</b></td>
    <td>50.98</td>
    <td>51.27</td>
    <td><b>59.37</b></td>
    <td>51.52</td>
  </tr>
  <tr>
    <td rowspan="2">mBERT</td>
    <td>Ori.</td>
    <td></td>
    <td>41.42</td>
    <td>44.99</td>
    <td>46.83</td>
    <td>45.54</td>
    <td>48.68</td>
    <td>46.92</td>
    <td>54.94</td>
    <td>47.05</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>41.42</td>
    <td>43.2</td>
    <td>44.77</td>
    <td>43.9</td>
    <td>47.2</td>
    <td>45.54</td>
    <td>50.86</td>
    <td>45.27</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>44.85</td>
    <td>47.02</td>
    <td>46.84</td>
    <td>45.4</td>
    <td>48.27</td>
    <td>46.88</td>
    <td>52.59</td>
    <td>47.41</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>44.85</td>
    <td>48.19</td>
    <td>47.61</td>
    <td>47.41</td>
    <td>48.84</td>
    <td>45.94</td>
    <td>52.24</td>
    <td>47.87</td>
  </tr>
  <tr>
    <td rowspan="2">BERT</td>
    <td>Trans.</td>
    <td>Siam.</td>
    <td>46.88</td>
    <td>49.95</td>
    <td>49.62</td>
    <td>52.16</td>
    <td>53.83</td>
    <td>48.62</td>
    <td>51.55</td>
    <td>50.37</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>45.66</td>
    <td>46.41</td>
    <td>48.69</td>
    <td>49.63</td>
    <td>48.87</td>
    <td>50.32</td>
    <td>52.76</td>
    <td>48.91</td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td>43.95</td>
    <td>47.73</td>
    <td>46.27</td>
    <td>48.33</td>
    <td>51.42</td>
    <td>49.18</td>
    <td>56.95</td>
    <td>49.12</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>43.95</td>
    <td>45.34</td>
    <td>45.58</td>
    <td>42.09</td>
    <td>52.85</td>
    <td>47.95</td>
    <td>53.28</td>
    <td>47.29</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td><b>48.94</b></td>
    <td><b>51.99</b></td>
    <td><b>55.41</b></td>
    <td>51.88</td>
    <td><b>56.14</b></td>
    <td>53.91</td>
    <td>52.3</td>
    <td><b>52.94</b></td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td><b>48.94</b></td>
    <td><b>53.04</b></td>
    <td>53.01</td>
    <td><b>52.56</b></td>
    <td><b>56.03</b></td>
    <td><b>55.36</b></td>
    <td><b>57.59</b></td>
    <td><b>53.79</b></td>
  </tr>
  <tr>
    <td rowspan="2">mDPR</td>
    <td>Ori.</td>
    <td></td>
    <td>45.19</td>
    <td>49.81</td>
    <td><b>53.08</b></td>
    <td>49.09</td>
    <td>51.65</td>
    <td>51.27</td>
    <td>55.98</td>
    <td>50.87</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>45.19</td>
    <td>48.01</td>
    <td>50.44</td>
    <td>48.08</td>
    <td>50.44</td>
    <td>52.44</td>
    <td>54.25</td>
    <td>49.83</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>45.59</td>
    <td>47.46</td>
    <td>47.74</td>
    <td>48.03</td>
    <td>49.33</td>
    <td>48.71</td>
    <td>53.51</td>
    <td>48.62</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>45.59</td>
    <td>46.81</td>
    <td>48.63</td>
    <td>47.93</td>
    <td>51.77</td>
    <td>48.42</td>
    <td>54.2</td>
    <td>49.05</td>
  </tr>
  <tr>
    <td>DPR</td>
    <td>Trans.</td>
    <td>Two-tow</td>
    <td>47.52</td>
    <td>50.07</td>
    <td>49.19</td>
    <td>52.55</td>
    <td>50.91</td>
    <td>50.56</td>
    <td>53.74</td>
    <td>50.65</td>
  </tr>
  <tr>
    <td rowspan="2">Ori.</td>
    <td>Ori.</td>
    <td></td>
    <td>43.2</td>
    <td>48.23</td>
    <td>51.69</td>
    <td>49.47</td>
    <td>45.59</td>
    <td>44.46</td>
    <td>53.62</td>
    <td>48.04</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>43.2</td>
    <td>42.41</td>
    <td>43.1</td>
    <td>39.87</td>
    <td>39.37</td>
    <td>39.83</td>
    <td>42.3</td>
    <td>41.44</td>
  </tr>
  <tr>
    <td rowspan="2">Siam.</td>
    <td>Ori.</td>
    <td></td>
    <td>41.64</td>
    <td>37.42</td>
    <td>46.07</td>
    <td>41.37</td>
    <td>53.79</td>
    <td>22.45</td>
    <td>53.45</td>
    <td>42.31</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>41.64</td>
    <td>44.8</td>
    <td>44.1</td>
    <td>45.92</td>
    <td>43.25</td>
    <td>49.92</td>
    <td>45.34</td>
    <td>45.00</td>
  </tr>
  <tr>
    <td rowspan="2">mLegal</td>
    <td>Ori.</td>
    <td></td>
    <td>46.28</td>
    <td>48.31</td>
    <td>51.87</td>
    <td>49.42</td>
    <td>50.96</td>
    <td>51.16</td>
    <td>54.6</td>
    <td>50.37</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>46.28</td>
    <td>48.53</td>
    <td>51.87</td>
    <td>49.42</td>
    <td>50.36</td>
    <td>50.38</td>
    <td>54.25</td>
    <td>50.16</td>
  </tr>
  <tr>
    <td rowspan="2">Two-tow</td>
    <td>Ori.</td>
    <td></td>
    <td>45.08</td>
    <td>48.18</td>
    <td>51.89</td>
    <td>46.19</td>
    <td>55.01</td>
    <td>47.99</td>
    <td>49.2</td>
    <td>49.08</td>
  </tr>
  <tr>
    <td>Trans.</td>
    <td></td>
    <td>45.08</td>
    <td>47.95</td>
    <td>52.47</td>
    <td>47.5</td>
    <td>55.14</td>
    <td>48.43</td>
    <td>53.56</td>
    <td>50.02</td>
  </tr>
</table>

Table 10: Recall@10% performance on unseen legal queries test split.
(a) All
(b) English
(c) French
(d) Italian
(e) Romanian
(f) Russian
(g) Turkish
(h) Ukrainian

Figure 3: LexCLiPR Data Analysis: Total Number of Paragraphs per Judgement

(a) All
(b) English
(c) French
(d) Italian
(e) Romanian
(f) Russian
(g) Turkish
(h) Ukrainian

Figure 4: LexCLiPR Data Analysis: Percentage of Relevant Paragraphs per judgement.

(a) All
(b) English
(c) French
(d) Italian
(e) Romanian
(f) Russian
(g) Turkish
(h) Ukrainian

Figure 5: LexCLiPR Data Analysis: Number of Tokens per Paragraph
(a) All                (b) English                (c) French                (d) Italian
(e) Romanian           (f) Russian                (g) Turkish               (h) Ukrainian

Figure 6: LexCLiPR Data Analysis: Number of Tokens per Query

Reviewer paper summaries: - This paper introduces LexCLiPR, a cross-lingual dataset for paragraph-level retrieval from European Court of Human Rights (ECtHR) judgments.

- The proposed dataset spans 7 languages using ECtHR case law guides and distance supervision.

- The evaluations are conducted via retrieval models in zero-shot and fine-tuned settings, and the performance analysis cover across languages and cross-lingual transfer scenarios
Summary of strengths: 1. The proposed LexCLiPR dataset is novel and valuable. It addresses a significant gap in the field of Legal IR by providing a cross-lingual resource for paragraph-level retrieval. 

2. The paper presents a thorough evaluation of various retrieval models, including zero-shot and fine-tuned approaches, monolingual and multilingual models, and different architectures. 

3. The task of paragraph-level retrieval from legal judgments is highly relevant to legal professionals, potentially saving significant time and improving efficiency in legal research.
Summary of weaknesses: 1.While the paper positions LexCLiPR as a novel contribution, it lacks a comparison to existing legal IR datasets (e.g. [1] & [2]). it would be beneficial to see a more detailed comparison with existing legal information retrieval datasets, even if they are not cross-lingual. This would help readers better understand the unique contributions of LexCLiPR.

2. The adopted evaluation metric is too limited. [1] and [2] use P@K, NDCG@K, MAP as the evaluation metrics, but the authors of this submisson only adopt R@K as the evaluation metric, which can cause biased evaluation results. The authors are expected to provide a more comprehensive evaluation for different models and architectures.

[1] LeCaRD: A Legal Case Retrieval Dataset for Chinese Law System, SIGIR 2021

[2] MUSER: A Multi-View Similar Case Retrieval Dataset, CIKM 2023
Comments, suggestions and typos: See weakness.
Overall assessment scores: 2.5