CaseEncoder: A Knowledge-enhanced Pre-trained Model for Legal Case Encoding
Authors: Yixiao Ma
Yueyue Wu
Weihang Su
Qingyao Ai
Yiqun LIU
Venue: EMNLP 2023 Main (2023)
Keywords: Legal case retrieval
pre-trained language model
knowledge
TL;DR: A pre-trained language model enhanced with legal knowledge to effectively encode legal documents.

Abstract: Legal case retrieval is a critical process for modern legal information systems. While recent studies have utilized pre-trained language models (PLMs) based on the general domain self-supervised pre-training paradigm to build models for legal case retrieval, there are limitations in using general domain PLMs as backbones. Specifically, these models may not fully capture the underlying legal features in legal case documents. To address this issue, we propose CaseEncoder, a legal document encoder that leverages fine-grained legal knowledge in both the data sampling and pre-training phases. In the data sampling phase, we enhance the quality of the training data by utilizing fine-grained law article information to guide the selection of positive and negative examples. In the pre-training phase, we design legal-specific pre-training tasks that align with the judging criteria of relevant legal cases. Based on these tasks, we introduce an innovative loss function called Biased Circle Loss to enhance the model's ability to recognize case relevance in fine grains. Experimental results on multiple benchmarks demonstrate that CaseEncoder significantly outperforms both existing general pre-training models and legal-specific pre-training models in zero-shot legal case retrieval. The source code of CaseEncoder can be found at https://github.com/Anonymous-EMNLP2023/CaseEncoder.

CaseEncoder: A Knowledge-enhamed Pre-trained Model for Legal Case Encoding
Yixiao Ma"
Yueyue Wu
Huawei Cloud BU
Quan Cheng Laboratory
Shenzhen, Guangdong, China Institute for Internet Judiciary, Tsinghua University
ma-yx wwwpettsiinuaa.rr..oom
DCST, Tsinghua University wuyueyue 6000gmail. com
Weihang Su
Qingyao Ai
Quan Cheng Laboratory
Quan Cheng Laboratory
DCST, Tsinghua University Institute for Internet Internet Judiciary, Tsinghua University
Institute for Internet Judiciary, Tsinghua University DCST, Tsinghua University swh22@mails. tsinghua. edu. cn
aiqingyad@gaiil.cm
Yiqun Liu
Quan Cheng Laboratory
DCST, Tsinghua University
Institute for Internet Judiciary, Tsinghua University yiqunl iu@tsinghuaunaa...moaa htwptssiinnuuuuueuuc.. edu.cn
Abstract
demonstrate that CaseEncoder significantly out- performs both existing general pre-training models and legal-specific pre-training models in zero-shot legal case retrieval. The source code of CaseEncoder be found https: //github. com/myx666//aalacaacccmc CaseeEnooer..
Legal case retrieval is a critical process for mod- ern legal information systems. While recent studies have utilized pre-trained language mod- els (PLMs) based on the general domain self- supervised pre-training paradigm to build mod- els for legal case retrieval, there are limitations in using general domain PLMs as backbones. Specifically, these models may not fully cap- ture the underlying legal features in legal case documents. To address this issue, we propose CaseEncoder, a legal document encoder that leverages fine-gna grained legal knowledge in both the data sampling and pre-training phases. In the data sampling phase, enhance the qual- ity of the training data by utilizing fine-grained law article information to guide selection of positive and negative examples. In the pre- training phase, we design legal-specific pre- training tasks that align with the judging crite- ria of relevant legal cases. Based on these tasks, we introduce an innovative loss function called Biased Circle Loss enhance the model's abil- ity to recognize case relevance in fine grains. Experimental results on multiple benchmarks
Introduction
Legal case retrieval is a critical process for modern legal information systems, as aims to find relevant prior cases (i.e., precedents) that serve as important references to the case to be judged. In recent years, pre-trained language modeling (PLM) techniques have achieved great success in general-domain trieval tasks, which has also led to attempts to apply PLMs in the legal domain. For example, Zhong et al. (2019) and Xiao et al. (2021) propose legal- oriented PL. based on BERT (Devlin et al., 2018) and Longformer (Beltagy et al., 2020), respectively. However, existing legal- oriented PLMs have lim- itations in their adaptation to the legal domain, because they mainly rely replacing general- domain training data with legal data extending the input length fit the long-length characteris- tic of legal documents. Ge et al. (2021) parse law articles in the form of train a multi-level matching network for legal case
This work done studying Quan Cheng Lab- oratory & Institute for Internet Judiciary, DCST, Tsinghua University Corresponding Author
pairs

---

matching, Bhattacharya al. (2022) propose Hier- SPCNet which substantially improves the network- based similarity by introducing legal textual infor- mation. However, they still adopt general-domain PLMs as the foundation of their approach. While interpretable, such models do not fully enable the PLM to comprehend legal concepts in case docu- ments, which limits their generalizability and ap- plicability.
as fine-grained legal knowledge can be applied all statutory law systems, or even to the common law systems, because statutory law is one of the most important sources of common law. Therefore, the effectiveness of the proposed methodology in this paper is not limited to a specific legal system. Experimental results on multiple legal case re- trieval datasets demonstrate that CaseEncoder sig- nificantly outperforms the existing general pre- training models and legal-specific pre-training mod- els. We also present case document embedding visualizations to showcase the potential of Cn-eLn- coder in downstream tasks such as charge predic- tion and article prediction.
To address this issue, this paper proposes CaseEncoder, PLM that leverages fine -ra-gra legal knowledge to improve both the data sampling phase and the pre-training phase. In the data sam- pling phase, we split law articles into unambiguous branches and construct inner logical relations in each article, which are used to estimate similarity weights between cases. These similarity weights serve as pseudo-labels to guide the selection of positive and negative cases. In this way, our pro- posed data sampling method improves the quality of sampled cases for further pre-training. In the pre-training phase, CaseEncoder adopts masked language modeling (MLM) and fine- -eeeggrrind con- trastive learning tasks. These tasks aim to match two main concepts in the judging criteria of legal relevance: key circumstances and key elements (Ma et al., 2021b), respectively. Key circumstances re- fer to significant case descriptions in the document, while key elements are the legal-Ievel abstraction of key circumstances, which focus more on consis- tency with law articles. In practice, the MLM task captures the semantic-level case descriptions, while the fine-grained contrastive learning task adopts sampled cases together with their relevance weights to enhance the model's ability to identify key el- ements. Based on the design of the fine iin-igrined contrastive learning task, we propose an innovative loss function, Biased Circle Loss, which leverages the obtained fine-grained relevance score to opti- mize the recognition of key elements.
2 Related Work
The methods for legal case retrieval can be divided into two categories: traditional bag- of-words methods and neural-based methods. Before the emergence of deep neural net- works, TF-IDF+VSM (Salton and Buckley, 1988). BM25 (Robertson et al., 1995), and LMIR (Ponte and Croft, 1998) are representative bag-of Pag-of-words methods. These methods have one common fea- ture: they all treat documents as sequence of terms and compute the document similarity score by statistical factors like term frequency or inverse document frequency. One limitation of such meth- ods is that they ignore the information of the order among terms.
With the rapid development of NLP techniques. applying pre-trained language models (PLMs) to le- gal case retricval has received huge success. West- ermann et al. (2020); Shao et al. (2020) directly retrieve legal cases through a fine-tuned BERT (De- vlín et al., 2018). Zhong et al. (2019) and Xiao et al. (2021) use large Chinese legal corpus to pre-train BERT and Longformer, respectively. In addition to applying original PLMs the legal domain, there are also works trying to incorporate legal knowl- edge into their proposed models. For example, Sun et al. (2022) propose Law-Match, a model-agnostic causal learning method that utilizes the knowledge in law articles to tackle legal case matching. Bar- ale et al. (2023) integrate the retrieval, processing, and information extraction into pipeline for the recognition of key information in refugee cases. Their model retrieves a total of 19 categories of items from refugee cases before conducting in- formation extraction by the state-of-the-an neural lamed-entity recognition method. Bhattacharya
There are two advantages of annotating law ar- ticles as fine-grained legal knowledge. 1) Cost efficiency: law articles are usually close sets with a limited number of documents regardless of the le- gal system, and the number of law articles is much smaller than the number of legal cases in reality. Therefore, annotating law articles requires much less effort than annotating legal cases. 2) Gener- aliability: legal knowledge from articles is appli- cable to all cases under the corresponding legal system. Notably, the idea of annotating law articles

---

of Hosmbous Drivigg
primarily on retrieving cases under criminal law, To match the real legal scenario that the retrieval model might have never seen g or its related cases before, in this paper, all models are evaluated in a zero-shot manner.
Court: People's Count XXXX Zonz, Standong Province
Case Numbor::Crinnll Criminal Indimment (2015) (XXX) Oriminal No. XXX
Parties' Information
Public Prosecuter: Shinding Province XXX People' Procumsoriee
Defendant Background: Delesdan XXX, male, bom
(Acousation of Prosecutor) On Decemteer 18, 2017, defendant was drunk and drove odinryy two-wheoled motoecycle along Fushan Road Weihai
City whtn a collited with a small cit...
(Evidlnns) .. and provied the following: 1. The amos iseual by the Public
Securiy Burezz of XCXX 2. Testimony of witnes XXX...
Method
(Facts Confirmed the Court) After inveeigatios, you found that drunk and
December 2017, defendant
The court believes
defendant
XXX siolated the transpertaion management
This section outlines the design and implementa- tion of CaseEncoder. Figure illustrates the overall framework of the model. We begin by introduc- ing the fine-grained case sampling method used for data preparation Section 4.1. Then, in Sec- tion 4.2, we describe the pre-training tasks pro- posed in CaseEncoder.
cegulations drove moter velicle on drunk, behavior
consunuted the crime of hazandous draving...
Decision:
Thee defendant XXX quilty hazardous driving, seatenced to ene mooth and
filuann days of detention...
Judge XXX
Morch 30, 2018 XXX
Figure 1: An example of legal case document.
MLM
al. (2022) incorporate domain knowledge-based document similarity into PCNet (Minocha al., 2015) and propose a heterogeneous network called Hier-SPCNet for a better representation of doc- ument similarity. However, these neural-based -bsed works all require task-specific fine-tuning or align- ment to reach relatively good performance. By comparison, our proposed method can be directly applied to different downstream tasks in the zero- shot setting.
Docs with weight Info
Pre-train Data
CascEncoder
Eyen + Faci
Learning
Fine arained
Sampling
Back propagation
Corpus
Figure 2: The overall framework of CaseEncoder.
4.1 Fine-grained Case Sampling
Recent studies on legal-oriented PLMS do not fo- cus on understanding legal texts from a bottom- up approach. In other words, most PLMs simply replace the general-domain training corpus with legal texts without considering the legal correla- tion between these texts. This is mainly because the annotation of legal cases is time-consuming and requires much expertise, making it challenging to collect large-scale labeled data. On the other hand, in contrastive learning, which has proven to be effective in the pre-training phase, data needs to be sampled in advance positive and negative cases. Unlike the general domain, in the legal do- main, it is not appropriate enough to sample the positive and negative legal cases simply based on raw information in documents (e.g., charges, law articles, etc.). Because in a real legal scenario, a judge usually adjudicates a case based on how well key circumstances and key elements of the case match the constituent elements ffie-ggaiiee inter- pretation of the law article). Therefore, this paper proposes fine- finengrained sampling method for legal case documents with reference to the specific pro- cess of judges deciding cases. By doing so, the sampled positive and negative cases can match the manually labeled relevance as much as possible for the subsequent contrastive learning task.
3 Task Definition
Given a query case the task aims to re- trieve relevant cases from a candidate list L = (c1, cM }. where M is the size of L, and rank them by the relevance to q. Each candidate case document in the list has three main components: 1. Facts are objective fact statements confirmed by the court based on the evidence provided by the defendant and plaintiff. These statements typically answer questions such as where, when, and how the case occurred. 2. Holding is the judge's opinion on the key arguments of the case. It explains the reasoning behind the judge's decision. 3. Decision contains the final judgment of the defendant in- cluding the charge, sentence, and articles involved, This component is the official outcome of case. An example of a legal case document is shown in Figure 1.
In the real legal case retrieval scenario, q is usu- ally a case to be judged. Therefore, q only con- tains the Facts of case, while candidate cases are precedents with a complete document including a title, meta information, Facts, Holding, Decision, and related law articles. In this paper, we focus

---

Articke 183-1: Dargenous Briving Offine Wheever drives muter velick coad and onnnts oscal the following octs shall be sestenced ioprestment and shall also be fined
Phrases
Intentionally raing with others win ciroustneees Dal are partioutarly secios
Branches
Mhlle Irunk
Delves roote
velicle on a nod will willedrua drank"
(1) imantisnnal racing wilk othen with cirunssnnces tht are parteslally serioss;
Enszaing sbool bus services
Inseniionaly excoolins the ruasinuu nurbher passengens
Kambitee
the masimur number of prosuggrs il
Thivve moter vebicle airee, excering
cruzzjnng passuuer mussportaiin nices
(2) Drovnn meter vchicle while drank:
Spdt
Driver a noteer velicle on read
Uatentionally
(3) Euszaing achool paweeace iomoporation paeeenge pisanited limil
Enaaging gassenger Insopettaioon servicea
Incetionally exceoding the prese hed speed with eehoss comceaacioos
Sog?
diivve moter vciile, noa, peregger
transpottaion, exceed, maximun run pessongess }
ciscepamas;
()) Viskenng salety isoandos chemicats incerionally teroperting
regsbriinnel of
Vislyingg the srley ranageeeet regelimens basordres chemeals moneonslly by mengporing
Seg?
I diive, nove velicle, nod, drink
locandes chonical wiich crdbegee public safety.
bscondee chemeale which endenger public valety
Corpus of Sequences
Figure 3: An illustration of the process of collecting fine-grained sequence corpus.
Generally, a law article covers multiple branches. A branch describes a specific and unambiguous situation that conforms to the article. For exam- ple, Article 133-/ from the Chinese Criminal Law shown in Figure 3 has four acts, among which certain branch each. but act (3) uses two 'or' clauses cover all branches. Specifically, 'engaging in school bus service' and 'engaging in passenger transportation service' belong to different branches and will never appear in the same case even if they belong to the same article. In this paper, we call such phrases in a parallel relationship. According to the multiplica- tion principle, act (3) describes 2 x 2 = 4 branches. branches
same branch, while phrases in a parallel relation- ship belong to different branches. Finally, we split each branch into words and remove all meaningless words (e.g., stopwords, prepositions). The rest of the words in one branch are called sequence. All such sequences belonging to the same law article constitute an article corpus. In conclusion, law system with N articles has N article corpus:
act (1), (2) and (4) deseribes
C = #CCrtticte##1 Carticle .... Cartidle#N (1)
Carticie#p {seqpl, seqp2, seqpT) P E [1
(2)
where T is the number of sequences (i.e., the num-
ber of branches) in article#p.
Therefore, Article 133-/ has 1 x 3 + in total.
With the help of article corpus, we can obtain fine-grained legal features of cases. Specifically, we first extract Holding from the case document, which contains reasons supporting the final judg- ment and is highly related to the articles of the case. In the legal domain, cases committing the same crime have articles in common. Therefore, our case sampling strategy can recognize their fine- grained legal similarity with the help of article cor- pus. Suppose case &; belongs article and the extracted Holding of case & is denoted as h,, we can compute the fine- finengrained legal feature Upp by:
According to Article 133-1, phrases in one arti- cle can be written not only sequentially, but also in a parallel way. In other words, law article is com- plexly structured. Even for two cases belonging to the same article, is hard for models recognize their legal relevance without any detailed informa- tion because such cases may belong to different branches of this article. To tackle this challenge, article information in legal cases requires further exploration in fine grains. In this paper, we iden- tify more fine-graines article information, namely branch-leved similarity, for each case the prelim- inary of finding positive and negative cases in our case sampling algorithm.
Uip [f hi), fS0002, h), f(eggerr h)] E IR"
(3)
where [(SCppp h2) denotes the BM25 (Robertson et al., 1995) score between the q-th sequence of article#p and h2 given corpus C.. The reason for using BM25 here is that experimental results show little difference between traditional and care- fully designed complex methods in representing the term-level similarity. terpreted as one feature of legal case at the fine- grained article level. Next, given two cases c; and
To get branch-level similarity of legal cases, we first need to extract branches from all law articles. As described in Figure 3, the extraction and pre- processing include the following steps: First, all law articles are split into phrases. Then, the phrases belonging to the same article are reconstructed into branches. Similar to the example of Article 133-1, phrases written sequentially belong to the
Vip can be intuitively in-

---

Cj. the fine-grat fine-grained similarity weight between two cases Wij can be presented as:
the idea of the judging criteria, CaseLincoder is pre-trained by two tasks for the recognition of key circumstances and key elements, respectively.
NAy x max(cos((iie Ugb) wij
|A
4.2.1 Pre-training task
article#k E A, A,)
(4)
The first pre-training task is the masked language modeling (MLM) task, which enables the capture of regular semantic-level meaning of case descrip- tion. As discussed Devlin et al. (2018); Liu et al. (2019); Ma et al. (2021a) la), MLM contributes to pro- ducing embeddings with contextual information. Such embeddings are beneficial to the representa- tion of key circumstances in legal cases. In detail, only select Facts in case document to domly mask 15% of the tokens for MLM, because key circumstances are all included Facts. The masked text is then fed into CaseEncoder to predict the masked tokens based on the surrounding un- masked tokens. The MLM loss function is defined as:
where A, is the set of articles involved in case Gis cos is the cosine similarity score, and k repre- sents the k-th article in As. In other words, the fine-graine -ne-ggained similarity weight between two cases is mainly determined by two aspects: the extent of article overlap between two cases and the cosine similarity between their fine-grane fine-grained legal feature ogk and vyk. Note that maximum function is added Equation 4, as Wij is valued by the most similar situation of all articles in A, n Aj-
Finally, for each case, G the legal case cor- pus, the fine-graing sree-gaaned sampling method can sample a legally explainable positive case C++ according to the value of Wh++, which can be regarded as pseudo label indicating legal relevance. Since the ume-grained legal feature may be helpful to pre- training, the input data for the pre-training phase will be in the form of quadruples:
LMLM =
Σ logp (a' X(m(x))
(5)
x'Em(x)
where X is the text in Facts, m(x) is the set of masked tokens, and X)(xx) the set of unmasked tokens.
(Cis Ct y Vip: U/
where p denotes the index of the law article with the maximum cos(Ci; ojs) value in Equation 4.
4.2.2 Pre-training task
4.2 Legal-specific Pre-training Task
The second pre-training task is a fine-grained con- trastive learning task that utilizes the information from quadruples (c,, 1 My 0/1) obtained in Sec- tion 4.1. The contrastive learning task in previous work (Chen et al., 2020) trains a model using aug- mented positive cases and regards the rest of the cases in the same batch as negatives. However, in the legal domain, the relevance scale is more fine- grained. One legal case be partially relevant another, and the extent of relevance is mostly deter- mined by the previously mentioned key elements. Therefore, a fine- ane-grained contrastive learning task is proposed to enhance the recognition of key el- ements. Specifically, suppose the batch size is N and each quadruple has two cases, the total number of cases in batch is 2N.. First, multi-layer Trans- former is adopted to obtain the representations of 2N cases. Then, we take the output of [CLS) token in the last hidden layer of Transformer as the case embedding: C1, C2, ..., C2N, G TR where H is the hidden size. Finally, the training objective this fine- grained contrastive learning task, Biased
As law-oriented pre-trained model, we aim to integrate legal knowledge into the design of pre- training tasks, so that the model can acquire the ability to understand case documents not only at the semantic level but also at the legal concept level after training. To this end, in this paper, we refer to the judging criteria of relevant cases to design our pre-training tasks. As demonstrated by Ma al. (2021b), two cases are relevant if they satisfy two requirements: high similarity between their key cir- cumstances, and high similarity between their key elements. Specifically, key circumstances refer to important case descriptions, while key elements fo- cus more on the consistency with law articles and represent the legal-level abstraction of key circum- stances. In summary, case is considered relevant another when the case description and abstracted legal concept are both relevant.
Considering the overall performance in other NLP tasks, we adopt RoBRTa (Liu et al., 2019) as the backbone of Encode.. Following

---

Circle Loss (BCL), defined as:
LeCaRD (Ma et al., 2021b) is the first Chinese le- gal case retrieval benchmark, which is widely used for the evaluation of retrieval models. Challenge of AT in Law (CAIL) (Xiao et al., 2018) 8) is a com- petition held annually to promote Al technology and higher level of digital justice since 2018. CAIL2021-LCR and CAIL2022-LCR are two (6)
LBCL = loggg11 [ + Σ exp(770 An))
K
exp( -a [p))]
a = exp wy -Op - s/p. a 1sh - Onl+
competition datasets of CAIL.
Notably, since the experiment setting is zero- shot as introduced Section no training or fine- tuning stage required for all methods mentioned in this paper. Therefore, both the training set and test set in the datasets are included for the evalua- tion. The total number of law articles involved in these datasets is 537. As retrieval task, all models in this paper evaluated by the Normalize Dis- counted Cumulative Gain (NDCG) metric. In this paper, we consider three different NDCG metrics: NDCG@ 10, NDCG@20, 20, and NDCG@30.
where Sp and 8n are cosine similarity scores be- tween positive and negative case embedding pairs, respectively. By and On are parameters control- ling the speed of convergence, where ap is deter- mined by the legal-specific relevance weight in Equation 4. 7. Op, On, Ap. and An are hyper- parameters scale factor, optimum for Sp, opti- mum for 8n, between-class margin, and within- class margin, respectively.
this way, CaseEn- coder is trained to pull case embeddings the same class closer and push case embeddings in different classes apart. The distance between case embeddings in the vector space depends on the value of 8p and 8u-
5.2 Baselines
For a comprehensive evaluation of CaseEncoder on legal case retrieval, in this paper, we adopt the following PLMs which are widely applied in the legal domain as baselines:
There are two main differences between our pro- posed loss function and Circle Loss (Sun et al., 2020): First, we expand the original loss function from a binary setting to a multi-class setting since legal cases within a batch can be classified into mul- tiple classes. In detail, we consider any two cases to be in the same class if their legal-specific rele- vance weight is larger than particular threshold Wr. and such rule is transitive across all cases in batch. Therefore, the actual implementation of Equation 6 more complicated because cases are of multiple classes. Sccond, we add a weight parameter a to LBCI. to account for the extent of relevance between cases. As a result, the optimiza- tion object will shift depending the value of w. By taking the legal-specific relevance weight into consideration, CaseEncoder is trained to diserimi- nate between relevant cases in fine grains.
BERT-XS (Zhong et al., 2019) is new lease of BERT (Devlin al., 2018) which conducts a secondary pre-training on Chinese criminal documents. Compared to the original BERT, the BERT-XS mainly focus on legal domain-specific tasks.
Lawformer (Xiao et al., 2021) the first law-oriented and longformer based (Beltagy et al., 2020) language model, which combines task-motivated global attention and local slid- ing window attention to capture long-distance information. Lawformer extends the input length from 512 tokens to 4096 tokens adapt the long length of legal documents.
Finally, Caseencoder is optimized by the linear combination of MLM loss and BCL loss:
BERT-PLI (Shao et al., 2020) aggregates paragraph-level semantic similarity between two case documents to retrieve legal cases. As the backbone of BERT-PLI, a BERT model is fine-tuned by case law dataset in COL- IEE2019 (Rabelo et al., 2019) for alignment.
Cioall LMLM ALBCL.
where À is hyper--aramettr
5
Experiments
5.1 Datasets and Evaluation Metrics
lhttp://cail.cpppa cipsc.orgg org...org..o or...rg..c................... wwgg..n//tssie html PraceID338 cail Sail_tag=2022
This paper adopts three publicly available datasets: LeCaRD, CAIL2021-LCR, and CAIL2022-LCR.
'http://cail. cipsc. waasc.org... ong..cn/fee ttt./ttkksssmmms t.mmmithhtin html?
racelD=1&cail_ tag==2221

---

Table 1: Evaluation results of different models on LeCaRD, CAIL202I-LCH, and CAIL2022-LCR, f\/ denote statistical significance compared CaseEncoder level of p-value < 0.05%.0.1..0000 using Wilcoxon test.
LeCaRD
CAIL202I-LCR
CAIL2022-LCR
Model
NDCG @ (10, 20, 30)
NDCG @ (10, 20, 30)
NDCG (CCG)@(1)) (10, 20, 30)
BERT-XS
0.343
0.355
0.384
0.361
0.368
0.388
0.358
0.359
0.383
BERT-PLI
0.525
0.519
0.538
0,555l
0.547$
0.560
0.512
0.499$
0.516
Lawformer
0.620
0.623
0.636
0.691
0.684
0.699
0.694
0.688
0.700
RoBERTa
0.748t
0.762
0.791
0.804t
0.817
0.850t
0.793
0.803
0.837t
RoBERTa-Legal
0.742t
0.7643
0.806
0.814
0.823
0.855
0.800
0.811
0.846
OpenAl API
0.672$
0.695$
0.729$
0.715
0.726
0.7655
0.7148
0.7253
0.7633
CaseEncoder 0.785
0.803
0.839
0.842
0.849
0.876
0.833
0.840
0.867
RoBERTa (Liu et al., 2019) is replication of BERT (Devlin al., 2018) which exten- sively studies the setting of paper-parameters and training data size. Experimental results on various NLP tasks show that RoBERTa can match or exceed the performance of many previous PLMs. Since all datasets in this pa- per are based on the Chinese law system, adopt the Chinese version of RoBERTa (Cui et al., 2019) as the baseline.
BM25 algorithm is implemented by Gensim (Re- hürck et al., 2011). The hyper-parameter for CaseEncoder is: = 16, Op 1.25, O, = 0.25, 6p = 0.75, 8n = 0.25, WT = 0.25, and cxp* 10 6 All training and experiments are con- ducted on eight 32G NVIDIA V100 GPUs.
5.4 Experimental Results
The overall performance of CaseEncoder and baselines on LeCaRD, CAIL202I-LCR, and CAIL2022-LCR datasets are shown in Table 1. Through the experimental results, we have the fol- lowing observations:
RoBERTa-Legal is legal version of RoBERTa that conducts secondary pre- training legal data using the MLM task.
Compared to baselines, CaseEncoder has the best results in terms of all evaluation metrics on three datasets. Besides, Caseencoder outper- forms baselines with statistical significance ex- cept for one result (i.c., NDCG @ 30 of RoBERTa- Legal CAIL2021-LCR) but still by large margin (i.e., 0.855 to 0.876). This phenomenon illustrates the effectiveness CaseEncoder in re- trieving legal cases without any fine-tuning. The mhowledge-enhaneee sampling strategy and pre- training tasks facilitate the retrieval of relevant cases with legal meaning.
OpenAl 3 is an official and publicly avail- able way to get text embeddings generated by OpenAI LLMs. In this paper, we adopt the second generation of embedding mod- els provided by OpenAI, sext-embedding-ade 002 (Brown et al., 2020), to measure the simi- larity of legal documents.
5.3 Implementation Details
For a fair comparison, CaseEncoder and basclines are adopted to retrieve legal cases in the same man- ner. Specifically, all models adopt a commonly used dual-encoder paradigm to retrieve legal cases. That is, models take the full text of query and Facts part of candidates separately to generate document- level query embeddings and candidate case embed- dings without any secondary pre-training or fine- tuning. The final retrieved ranking list sorted by the cosine similarity between query embeddings and candidate case embeddings.
The overall results LeCaRD comprehen- sively lower than those on CAIL202I-LCR and CAIL2222-LCR datasets. This is mainly due to the heterogeneity of the data corpus. Specif- ically, the pooling of candidate documents LeCaRD does not have any restrictions, while CAIL2021 and CAIL2022-LCR filter out documents with extremely long length. Conse- quently, LeCaRD is more challenging than the other datasets.
All PLMs are imported from Huggingface (Wolf et al., 2019), with the learning rate set to 10-5
As the backbone of CascEncoder, RoBERTa has competitive results among all baselines, which is
Chttps://platfrt ieeaataaamooop.iii.....mmm httprr/..paaa..o////.oo com/docs/guides embeddings

---

Table 2: Ablation study on LeCaRD. "w/o sampling' denotes CaseEncoder without the fine-grained sampling method. 'w/o BCL (infoNCE) and 'w/o BCL (Circle)' denotes replacing BCL with infoNCE and Circle Loss, respectively. 'w/o task' denotes CaseEncoder only pre- trained by MLM task.
Model CaseLnncoder w/o sampling
NDCG @ (10, 20, 30)
0.785, 0.803, 0.839
0.778, 0.795, 0.827
w/o BCL (infoNCE)
0.741, 0.764, 0.805
w/o BCL (Circle)
0.762, 0.782, 0.816
w/o task
0.742, 0.764, 0.806
consistent with its performance on other NLP tasks. Furthermore, RoBERTa-Legal has the second-best overall performance and outper- forms the original RoBERTa, which proves the idea in Gururangan et al. (2020) that a secondary pre-training using domain-speede -specin data is ben- eficial to the overall performance in the target domain. Finally, CaseEncoder outperforms its backbone and the legal version of its backbone, which indicates that the effectiveness of CaseEn- coder is not only due to the training data.
RoBERTa
Cavefinnnder (Curs
Bribery
Emberzlement
Provocation
Public Brawd Theft
Robbery
Figure 4: The visualization of case embeddings gener- ated by four PLMs in the zero-shot manner.
5.5 Ablation study
To investigate the effectiveness of our proposed fine-grained sampling method, the knowledge- enhanced contrastive learning task, and our pro- posed BCL loss function, we further conduct an ablation study. In detail, we: 1) replace our sam- pling method with normal in-batch sampling (cases belonging to the same article simply regarded as positive cases), 2) replace the BCL loss with in- foNCE (Oord al., 2018), 3) replace the BCL loss with Circle Loss (Sun et al., 2020) and 4) remove the fine-grained contrastive learning task, respec- tively. As shown in Table 2, replacing the sam- pling method, BCL loss function, or removing the contrastive learning task all lead to performance decline. Therefore, all of these innovations pro- posed in this paper contribute the effectiveness of CascEncoder. The result in Table 2 also indicates that BCL contributes most to the improvement of CaseFncoder. The traditional binary contrastive learning task has limited contribution to the per- formance because removing the whole contrastive learning task has similar performance to replacing BCL with infoNCE, which means the improvement is mostly made by BCL.
The effectiveness of BERT-XS (Zhong et al., 2019) is limited because it utilizes Next Sentence Prediction (NSP) task for pre-training and its [CLS] token not trained to represent document- level embeddings. Besides, BERT-PLI not as effective as reported in Shao et al. (2020). One possible explanation is that BERT-PLI utilizes the dataset from the entailment task in COL- external le-
IEE2020 (Rabelo et al., 2020) gal data source fine-tune the model. For fair comparison, no external data imported for any further training in this paper. Therefore, the per- formance BERT-PLI declines compared to its previous results reported Shao et al. (2020).
The embedding LLM provided by OpenAl is not as effective in zero-shot legal case retrieval as it is in the general domain. Its overall perfor- mance is even surpassed by PLMs with much fewer parameter scales such as RoBERTa and RoBeRTa-Legal. This result indicates that cur- rently, LLM in the general domain not the best choice for legal case retrieval.
5.6 Visualization of Case Embeddings
CaseEncoder is designed effectively model case documents in the legal domain. In addition to the retrieval task, the document-level case embedding

---

can also be utilized in other downstream tasks such as charge prediction, sentence prediction, and law article recommendation. Therefore, the quality of document-level case embeddings is the basis of downstream task performances. Figure 4 is an ex- ample of how CaseEncoder improves the quality of case embeddings for charge prediction. The cases in Figure 4 belong to six criminal charges. For each charge, randomly select 2,500 cases from the legal corpus and generate their case embed- dings using four different methods. Then, we use t- SNE (Van der Maaten and Hinton, 2008) to reduce the dimension of case embeddings for visualiza- tion. Among all PLMs. CaseEncoder has the best ability to divide case embeddings into six clusters based on their charges, with only one pair of sim- ilar charges (Provocation and Public Brawl) hav- ing some overlap. By comparison, RoBERTa par- tially distinguishes between six charges, but with more overlap than CaseEncoder. The performance of BERT-XS and Lawformer is limited, which is consistent with the retrieval result and explanation in Section 5.4. These visualizations demonstrate how the fine-grained legal knowledge embedded in CaseEncoder can be leveraged for range of legal applications beyond case retrieval.
systems and languages. We will also explore the potential of CaseEncoder for different downstream legal tasks. Secondly, the transitive rule in Sec- tion 4.2.2 is not consistent under all circumstances. For example, suppose case A and B are the same class, and B and C are in the same class, then A and C still might have no law articles in common if case A and C both belong to multiple classes. Ac- cording to experimental results, the influence of this special case can be alleviated by reducing the batch size in the pre-training stage, because the possibility of two multiple-clase cases appearing in the same training batch drops with the reduction of batch size. We will leave the optimization of our proposed BCL function as our future work.
A Acknowledgements
This work is supported by Quan Cheng Laboratory (Grant No. QCLZD202301) and the Natural Sci- ence Foundation of China (Grant No. 62002194).
References
Claire Barale, Michael Rovatsos, and Nehal Bhuta. 2023. Automated refugee case analysis: An nip pipeline for supporting legal practitioners. arXiv preprint arXiv:2305. 15533.
6 Conclusion and Limitation
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-documen transformer. arXiv preprint arXivv2004.05/500
This paper proposes CaseEncoder, a pre-trained encoder that utilizes fine-grained legal knowledge to enhance the representation of case document em- beddings. By introducing law article annotation into the sampling method, we improve the qual- ity of sampled positive and negative cases during data preparation. Next, the pre-training stage, Ensoerr adopts two legal-specific pre-training tasks to align with the relevance judgment criteria in the legal domain. Experiments and visual anal- ysis demonstrate the effectiveness of case embed- dings generated by CaseEncoderr in solving zero- shot legal case retrieval.
Paheli Bhattacharya, Kripabandhu Ghosh, Arindam Pal, and Saptarshi Ghosh. 2022. Legal case document similarity: You need both network and text. Informa- rion Processing & Management, 59(6):03369,
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam. Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In In- ternational conference machine learning, pages 1597-1607. PMLR
The main limitations of Caseercoder are: Firstly, the definition of similar legal cases is based on the assumption that "cases committing the same crime have articles in common", which is not applicable to all legal systems. However, the idea of anno- tating law articles as fine-grained legal knowledge enhance the performance of PLMs has the po- tential to be adopted in other law systems. In the future, we will consider validating the effective- ness of Encoerr on datasets of different law
Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziging Yang, Shijin Wang, and Guoping Hu. 2019. Pre- training with whole word masking for chinese bert. arXiv preprint 1rXiv:1906.08/0/
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint

---

Jidong Ge, Xiaoyu Shen, Chuanyi Li, Wei Hu, Bin Luo, et al. 2021. Learning fine-grane ine-grained fact-article correspondence in legal cases. arXiv preprint arXiv:2104. 10726.
Gerard Salton and Christopher Buckley. 1988. Term- weighting approaches in automatic text retrieval. In- formation processing & management, 24(5):513- 523.
Suchin Gururangan, Ana Marasovie, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't t stop pretraining: adapt language models domains and tasks, arXiv preprint arXiv:2004.1 10964.
Yunqiu Shao, Jiaxin Mao, Yiqun Liu, Weizhi Ma, Ken Satoh, Min Zhang. and Shaoping Ma. 2020. Bert-pli: Modeling paragraph-level interactions for legal case retrieval. In IJCAI, pages 3501-3507
Yifan Sun, Changmao Cheng, Yuhan Zhang. Chi Zhang. Liang Zheng, Zhongdao Wang, and Yichen Wei. 2020. Circle loss: A unified perspective of pair simi- larity optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pages 6398-6407.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1e...1997..992 1692.
Xinyu Ma, Jiafeng Guo, Ruqing Zhang. Yixing Fan, Xi- ang Ji, and Xueqi Cheng. 2021a. Prop: Pre-training with representative words prediction for ad-hoc re- trieval. In P Proceedings of the 14th ACM international conference on web search and data mining, pages 283-291.
Zhongxiang Sun, Jun Xu, Xiao Zhang, Zhenhua Dong, and Ji- Ji-Rong Wen. 2022. Law article-enhanced le- gal case matching: a model-agnostt causal learning approach. arXiv preprint a1Xiv:2210.11012.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Yixiao Ma. Yunqiu Shao, Yueyue Wu, Yiqun Liu, Ruizhe Zhang, Min Zhang, and Shaoping Ma. 2021b. Lecard: legal retrieval dataset for chinese law system. Information Retrieval (IR), 2:22.
Hannes Westermann, Jaromir Savelka, and Karim Benyekhlef. 2020. Paragraph similarity scoring and fine-tuned bert for legal information retrieval and entailment, In JSAI International Symposium on Ar- rificial Intelligence, pages 269-285.. Springer.
Akshay Minocha, Navjyoti Singh, and Arjit Srivastava. 2015. Finding relevant indian judgments using dis- persion of citation network. In Proceedings the 24th International Conference on World Wide Web, pages
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz. et al. 2019. Huggingface's transformers: State-of- the-art natural language processing. arXiv preprint arXiv:1910.03771.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv: 1807,03744
Jay Ponte and W Bruce Croft. 1998. A language modeling approach to information retrieval. In Pro- ceedings the s annual international ACM SIGIR conference on Research and development informa- tion retrieval, pages 275-281.
Chaojun Xiao, Xueyu Hu. Zhiyuan Liu, Cunchao Tu, and Maosong Sun. 2021. Lawformer: pre-trained language model for chinese legal long documents. Al Open.
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018. Cail2018: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv:1807.00 1807.024788
Juliano Rabelo, Mi-Youn, Young Kim, Randy Goebel, Masa- haru Yoshioka, Yoshinobu Kano, and Ken Satoh. 2019. A summary of the coliee 2019 competition. In JSAI International Symposium on Artificial Intelli- gence, pages 34-49. Springer.
Haoxi Zhong, Zhengyan Zhang, Zhiyuan Liu, and
Juliano Rabelo. Mi- Young Kim, Randy Goebel, Masa- haru Yoshioka, Yoshinobu Kano, and Ken Satoh. 2020. Coliee 2020: methods for legal document retrieval and entailment. In JSAI International Sym- posium on Artifiiill Intelligence, pages 196-210. Springer.
Maosong Sun. 2019. Open chinese language pre- trained model zoo. Technical report.
Radim Rehüřck, Petr Sojka, al. 2011. Gen- sim statisiicaa semantics python. Retrieved from genism. org.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.


Reviews:
[{"review_id": "EzSmnWSMPQ", "reviewer": "Reviewer_aaUB", "summary": "", "questions": "A) Line 406: since contrastive learning aims to distinguish positive pairs from negative ones, I wonder why the loss function should be in the multi-class setting.\n\nB) Line 408 to 411: this transitive rule seems to lead to inconsistency. For example, A and B are in the same class, and B and C are in the same class, but these two facts do not necessarily imply that cases A and C are in the same class via your rule.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Frist of all, thank you for your review! The review comments will substantially improve our work. Due to space limit, we focus on addressing the most important concerns:\n### 1. Lack of case study to show the uesfulness of key circumstances and key elements\n**A:** We have now added a case study to prove the usefulness. Due to space limit, we only show the main results:\n```\nGiven:\n- Query case id=3746\n- Doc id=23502, relevance label = 0 (negative)\n\nResults:\n- cosine(Embd_Q, Embd_D1) = 0.263\n- cosine(Embd_Q, Embd_D2) = 0.286\n- cosine(Embd_Q, Embd_D3) = 0.662\n```\nwhere `Embd_Q` is the query embedding, `Embd_D1, Embd_D2, Embd_D3` are doc embeddings generated by our full model, model without key circumstances, and model without key elements, respectively.\n\nWe will add this case study with more details to our camera-ready version.\n\n### 2. Why loss function should be multi-class?\n**A:** In fact, this is one of the main contributions of our work, because we realize the relevance between two legal cases can not be simply specified by 'positive' or 'negative'. Instead, there are different degrees of 'positive' or 'negative'. Our proposed BCL loss can help model recognize the fine-grained relevance, which pushes\\pulls doc embeddings to a different distance depending on their relevance to query embeddings.\n\n### 3. The transitive rule seems to lead to inconsistency?\n**A:** That is a really good question! We acknowledge that this problem does exist. However, according to our experiment, the transitive rule applys to most situations (>92%) where case A,B,C all belong to a single class. Even in the multi-class situation, there are only part of the cases where the transitive rule doesn't work, so the influence on the overall performance is very limited. We will leave it as the future work and discuss its influence in the experiment section.\n\n### 4. About the writting of sections\n**A:** Sorry for the confusing, we will reorganize and improve Section 4.1&4.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of splitting legal articles into branches is interesting, and I think it is potentially valuable for future studies on this task.\n2. The proposed branch-level similarity is a simple yet effective method to measure the relevance between cases and articles, which shows helpful to the contrastive learning process.\n3. Extensive experiments demonstrate the effectiveness of the proposed method.", "reasons_to_reject": "1. Although this paper mentioned two important kinds of concepts, i.e., key circumstances and key elements, they are indirectly modeled via two pretraining tasks. In addition, this paper does not provide any case study to show that the model indeed comprehends these two kinds of concepts better.\n2. Some parts of the model section are a little confusing.", "questions_to_authors": "A) Line 406: since contrastive learning aims to distinguish positive pairs from negative ones, I wonder why the loss function should be in the multi-class setting.\n\nB) Line 408 to 411: this transitive rule seems to lead to inconsistency. For example, A and B are in the same class, and B and C are in the same class, but these two facts do not necessarily imply that cases A and C are in the same class via your rule.", "paper_topic_and_main_contributions": "This paper focuses on legal case retrieval, which aims to find relevant legal cases for a specific case. It utilizes law articles for pretraining, which enables PLM to learn legal concepts in legal cases. Specifically, to sample positive and negative cases, it utilizes different branches to construct branch-level similarity for the contrastive learning task. Together with the traditional MLM task, the PLM is able to capture key elements and key circumstances information in the cases. The experimental results show the effectiveness of the proposed model.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Section 4.2 is too long. I suggest splitting it into two or three subsections."}, "extra_text": {}}, {"review_id": "UiYtTo6aKY", "reviewer": "Reviewer_E28B", "summary": "", "questions": "Do you have an explanation or a comment on the limitation on the similar case definition mentioned in the paragraph above? (I noticed there is no limitation section in the paper for now, it could be useful to add in the camera-ready and to answer this question there). ", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We do appreciate your recognition of our work! The suggestions are helpful to the refinement of our work. Due to space limit, we focus on addressing the most important concerns:\n### 1. The assumption depends on the type of law if civil or common\n**A:** Thank you for your suggestion! We acknowledge the assumption has limitations. However, in addition to civil law, our proposed method can also be applied to common law if the judgment basis in the assumption is switched from article to legal issues. Specifically, we need to first prepare common law data with annotated issues to pretrain our CaseEncoder, with the rest model design remains unchanged. In this way, the idea of recognizing relavant legal cases in fine grains is adaptive to different legal systems.\n\n### 2. To differentiate among cases that include the same \"crime\"\n**A:** Cases committed the same crime can still be differentiated because we focus not only on the overlap of legal articles, but also on the similarity of key judgment sentences in two case documents. In this way, relevant cases can be recognized in fine grains. The idea of sampling fine-grained cases is introduced in Section 4.1. We will add more explanations and details to this section in our camera-ready version.\n\n### 3. Missing limitation and references\n**A:** Thank you for your careful review! We will discuss the limitation of our work(including your first question) and add missing references in our camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- PLMs may not be a good fit for legal documents as it fails to capture their specificities, because their aim is to have a broad coverage. This is the starting point of the paper and the identified problem it aims to solve. This is very relevant to the current domain-specific NLP literature and concerns. The authors seek to re-balance this by introducing a layer of legal knowledge, and aiming for a more domain-targeted pre-training. Literature has indeed shown that models that are closer to a domain, eg legal, despite their smaller size, may still perform better on a range of tasks. \n- Very precise task definition in section 3\n- Section 4.1: the fine-grain case sampling aims at addressing the usual bottleneck of explicit supervision, using \"branch level similarity\". This is very relevant to the legal domain in particular that suffers from lack of resources.\n- The idea of introducing legal pre-training is interesting, while most legal NLP work have been focusing on fine-tuning PLMs. \n- Evaluation is made on several benchmarks of legal NLP and CaseEncoder always performs better. \n\nOverall, this is a very interesting paper, well written and clear, that explores a new way of introducing knowledge and of fitting the specific needs and challenges of legal NLP. ", "reasons_to_reject": "- Section 4.1: because the sampling is based on law article, with the aim of retrieving similar cases, the authors make the assumption that \"cases committing the same crime have articles in common\" line 277. One could argue that this may not always be the case, depending on the jurisdiction, the type of law if civil or common, and the area of law (e.g. international law). Another drawback is that it would not allow to differentiate among cases that include the same \"crime\". This rather limited and bounded definition of case similarity is a limitation. ", "questions_to_authors": "Do you have an explanation or a comment on the limitation on the similar case definition mentioned in the paragraph above? (I noticed there is no limitation section in the paper for now, it could be useful to add in the camera-ready and to answer this question there). ", "paper_topic_and_main_contributions": "This paper introduces CaseEncoder, an encoder designed specifically for legal documents that relies on introducing knowledge at the pre-training step and at the data sampling step. The targeted task is Legal Case Retrieval. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "\nAnother paper addressing past case retrieval, by means of targeted information extraction, that may be of interest to the authors: https://aclanthology.org/2023.findings-acl.187/"}, "extra_text": {}}, {"review_id": "aw86R2hBSv", "reviewer": "Reviewer_8osR", "summary": "", "questions": "- What's the typical length of a sample in each of the evaluation dataset? Most of the models mentioned in this paper rely on BERT or RoBERTa encoders, which have max_seq_len of 512 or 1024 BPE tokens. Is that enough for legal cases?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comments and suggestions, which we believe is beneficial to the refinement of our work.Due to space limit, we only address on the most important concerns:\n### 1. Need more IR models as baselines.\n**A:** That is a good suggestion! However, due to the time limit of this reply period, we currently compare our proposed CaseEncoder to one of the best models, ColBERT-v2, as you metioned, on three datasets in the zero-shot setting to demonstrate the effectiveness of CaseEncoder:\n|                    | **LeCaRD**        | **CAIL21-LCR**     | **CAIL22-LCR**    |\n|:----------------------:|:-----------------:|:------------------:|:-----------------:|\n|                | NDCG@(10, 20, 30) | NDCG@(10, 20, 30)  | NDCG@(10, 20, 30) |\n| **ColBERT-v2**         | 0.715 0.743 0.803 | 0.814 0.805 0.851  | 0.791 0.780 0.837 |\n| **CaseEncoder (Ours)** | 0.785 0.803 0.839 | 0.842 0.849 0.876  | 0.833 0.840 0.867 |\n\nThe result shows that CaseEncoder is comprehensively better than ColBERT-v2. Currently we are still doing experiments and the results of the rest baseline models will be reported in the camera-ready version of our paper.\n\n### 2. Reproducibility\n**A:** The source code of CaseEncoder together with checkpoint are available on https://github.com/Anonymous-EMNLP2023/CaseEncoder. All datasets used in this paper are public benchmarks for legal case retrieval.\n\n### 3. Rely on domain-specific techniques & not for NLP community\n**A:** Thank you for your question. We believe that legal domain is a very important part of NLP community. Currently various legal tasks (e.g., legal search, judgement prediction, legal text generation, and so on.) rely heavily on the NLP techniques. \n\nIn addition, we emphasize that the techniques proposed in this paper is not restricted to the legal domain. The idea of sampling fine-grained pre-training data, BCL, and the fine-grained contrastive learning task are general NLP techniques adopted in the legal domain instead of pure legal research. Our proposed method can be applied to other NLP domains which require a fine-grained recognition of unlabeled documents in the pre-training stage.\n\n### 4. Missing introcution of the pre-training data\n**A:** Thank you for your careful review! The pre-training data used in this paper are raw criminal judgment documents randomly sampled from online website, which is introduced in Section 3. However, we acknowledge that the current introduction is insufficient. We will add more details about the pre-training data in the camera-ready version.\n\n### 5. Is the model input length enough for legal cases?\n**A:** According to our statistics, the average length of legal cases is over 2000. However, in this paper, only the **fact** part (<700 words) of case documents is input into models, which is widely accepted and adopted in previous works. This is because **fact** is the key part to describe the case and is enough for the pre-training of language models. The rest of a case document is used in other places, such as the reference for fine-grained sampling, the extraction of legal articles. Therefore, the model input length is enough for legal cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed model consistently outperforms the RoBERTa baseline as well as other reported methods.\n\n- The ablations are helpful in that it shows how much impact each ingredient has on the final performance (e.g. data sampling, training objective, etc.).", "reasons_to_reject": "- This paper may not be suitable for the EMNLP venue for two reasons: 1) The proposed approach relies heavily on domain-specific techniques and the method is not general enough to provide much value for the wider NLP research community. 2) The reviewers may not have the necessary legal acumen to properly judge the claims of this paper. For instance, the authors justifies the generality of its main methodology by claiming that \"the idea of annotating law articles as fine-grained legal knowledge can be applied ... even to the common law systems. Therefore, the effectiveness of the proposed methodology in this paper is not limited to a specific legal system.\", something EMNLP reviewers may not be able to verify. This paper is probably better suited for a legal NLP conference or workshop.\n\n- The paper focuses on a \"retrieval\" task but only considers embedding models among its baselines. It would be great to add comparison to standard IR models. For instance, the BM25 baseline, SoTA dense retrievers (e.g. DRAGON https://huggingface.co/facebook/dragon-plus-query-encoder), SoTA sparse retrievers (e.g. SPLADE++ https://github.com/naver/splade), or multi-vector retrievers (e.g. ColBERT-v2 https://github.com/stanford-futuredata/ColBERT). Complex domain-specific approaches are better justified if they're shown to significantly outperform off-the-shelf retrieval models.\n\n- Some important details are missing. Please correct me in the rebuttal if I missed anything, but I failed to find any mention of the \"pre-training data\" used in this paper. It only mentions the evaluation datasets, but nothing is discussed on the data used to pre-train the model. What's the corpus for MLM? What's the case corpus used in the contrastive training?", "questions_to_authors": "- What's the typical length of a sample in each of the evaluation dataset? Most of the models mentioned in this paper rely on BERT or RoBERTa encoders, which have max_seq_len of 512 or 1024 BPE tokens. Is that enough for legal cases?", "paper_topic_and_main_contributions": "This paper studies the problem of legal case retrieval (finding relevant prior cases for a given case). It proposes domain-specific techniques to further pre-train RoBERTa with the MLM objective, as well as a contrastive objective tailor-made for legal cases. In particular, it modifies standard in-batch sampling with a sampling approach that uses domain expertise to measure the similarity between cases (e.g. how close two cases are consistent with the same law article). It then uses a modified version of Circle Loss (Sun et al., 2020) as the training objective, which is combined with the standard MLM objective.\n\nThe model is evaluated on three Chinese legal case retrieval datasets, and shows that it outperforms the RoBERTa baseline by 2 - 4 points. (Other baselines are also reported, but they often achieve much lower accuracy compared to the RoBERTa baseline.)", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers find the paper sound and exciting (3+ for both).  They note the novelty of the task and the clearness of task definition.  A noted concern is the broad applicability of this research methodology to outside of the legal domain, as well as other legal jurisdictions than the area studied.
Decision: Accept-Main