Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction
Authors: Leilei Gan
Baokui Li
Kun Kuang
Yating Zhang
Lei Wang
Anh Tuan Luu
Yi Yang
Fei Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: legal artificial intelligence
legal judgment prediction
contrastive learning
information extraction
TL;DR: we exploit contrastive learning and numerical evidence for confusing legal judgment prediction.

Abstract: Given the fact description text of a legal case, legal judgment prediction (LJP) aims to predict the case's charge, applicable law article, and term of penalty. A core problem of LJP is distinguishing confusing legal cases where only subtle text differences exist. Previous studies fail to distinguish different classification errors with a standard cross-entropy classification loss and ignore the numbers in the fact description for predicting the term of penalty. To tackle these issues, in this work, first, in order to exploit the numbers in legal cases for predicting the term of penalty of certain charges, we enhance the representation of the fact description with extracted crime amounts which are encoded by a pre-trained numeracy model. Second, we propose a moco-based supervised contrastive learning to learn distinguishable representations and explore the best strategy to construct positive example pairs to benefit all three subtasks of LJP simultaneously. Extensive experiments on real-world datasets show that the proposed method achieves new state-of-the-art results, particularly for confusing legal cases. Ablation studies also demonstrate the effectiveness of each component.

Exploiting Contrastive Learning and Numerical Evidence for Confusing Legal Judgment Prediction
Leilei Gan', Baokui Li', Kun Kuang",, Yating Zhang?, Lei Wang', Luu Anh Tuan', Yi Yang' and Fei 156
'Zhejiang University Allbbba Group University of Massachuselts at Amherst
"Nanyang Technological University SShangaai Al Laboratory Shanggaai Institute for Advanced Study of Zhejiang University
(leileigan, libaokui) @ zju.edd.ccn ranran.zyt @alibaba-inc.com aubuan.lu@@nu..cdu..s
Abstract
Fart Dearrpptien:
on Dexember 2014, celerder quancl with the plantilf boonse Insial mulers Laier, dettnlan plzntiil Accooding the Oonsak Hentifaazion.
Given the fact description text of legal case, legal judgment prediction (IJJ) aims to predict the case's s charge, applicable law article, and term of penalty. A core challenge of LJP is distinguishing between confusing legal cases that exhibit only subtle textual or number dif- ferences. To tackle this challenge, in this pa- per, we present framework that leverages MoCo-based supervised contrastive learning and weakly supervised numerical evidence for confusing I.P.. Firstly, to make the extraction of numerical evidence (the total crime amount) easier, the framework proposes to formalize named entity recognition task. Secondly, the framework introduces the MoCo-based su- pervised contrastive learning for multi-task LJP and explores the best strategy to construct posi- tive example pairs to benefit all three subtasks of LJP simultaneously. Extensive experiments on real-world datasets show that the proposed method achieves new statc-of-the-art results, particularly for confusing legal cases. Addi- tionally, ablation studies demonstrate the effec- tiveness of each component'.
XX his
Descriptin::
Begree nose Bijrry plamii noed mpnt
Mc 5MIB
Gau vietia wlik:
Charge Picking Quarrdl
costaieed cadl $2,000
Calbaa Article shall
Golden Teras
imgeloomeent
st monthe
Output LADANN
Charge Innentonaal Ingrry
Tera: seans
A Ginakomm Inpelommeni of dea
Articte 2N4
Whonver intentionally injao
another person drall be smmeneal
(a)
(b)
Figure I: Two examples of confusing legal cases. Fig- ure 1 (a) shows an example whose golden charge label Crime of Picking a Quarrel. which easily classified into its confusing charge Crime of Intention Injury. Fig- ure 1 (b) shows an erroneous prediction of the term of penalty, which does not exploit the amounts related to the crime.
with totally different charges, applicable law ar- ticles, terms of penalty. Figure 1 shows two examples of confusing legal cases. Figure (a) shows an example whose golden charge label is Crime of Picking a Quarrel, which is easily clas- sified into its confusing charge Crime of Intention Injury. Figure (b) shows an erroneous prediction of the term of penalty, which does not exploit the amounts related to the crime. A series of studies on this problem has been conducted, including manu- ally discriminative legal attributes annotation (Hu et al., 2018), distinguishing representations learn- ing via graph neural networks (Xu et al., 2020), and separating the fact description into different circumstances for different subtasks (Yue et al., 2021).
1 Introduction
Legal judgment prediction (LJP) is one of the most attractive research topics among legal artificial in- telligence (Cui et al.. 2022: Feng et al., 2022a; Zhong et al., 2020). Given the fact description text of a legal case, LJP aims to predict its charge, applicable law article, and term of penalty (Aletras et al., 2016; Zhong et al., 2018; Yang et al., 2019; Chalkidis et al., 2019; Gan et al., 202 Lyu et al..
2022; Liu et al., 2023; Zhang et al., 2023).
However, we argue that there exist two draw- backs to these studies. Firstly, all of these stud- ies use a standard cross-entropy classification loss, which cannot distinguish different mistaken classi- fication errors. For example, the error of classifying the charge Crime of Picking a Quarrel into its con- fusing charge Crime of Intention Injury is the same
One core problem hindering the performance of LJP from being satisfying is confusing legal cases, which have subtle text or number differences. but
'Our code is available at httpss///itthb. com/ leil Dasllegaan/Conrrstiivee "Carresponding Author.

---

classifying Crime of Picking a Quarrel into a not confusing charge Crime of Rape. The model should be punished more if it classifies a charge into its corresponding confusing charges. Secondly, the crime amounts in the fact description are eru- cial evidence for predicting the penalty terms of certain types of cases, such as financial legal cases. However, the crime amounts are distributed ran- domly throughout the fact description. Thus it is difficult for the model to directly deduce the pre- cise total crime amount and predict correct penalty terms based on the scattered numbers.
exists a Contrudictory Phenomenon. Contradic- tory Phenomenon means that instances with the same charge label may have different applicable law or penalty term labels. If we pair the training instances with the same charge label as positive examples, the resulting learned shared features will benefit the charge prediction task but will degrade the performance of the other two tasks. To tackle this challenge, we explore the best way to construct positive examples, which can benefit all three sub- tasks of L.PP simultaneously.
The proposed framework provides the following merits for predicting confusing LJP: 1) the frame- work is model -agnostic which can be used to im- prove any existing LJP models; 2) the extracted numerical evidence makes the predictions of the term of penalty more interpretable, which is crit- ical for legal judgment prediction; 3) compared with previous studies (Hu et al., 2018), the use of supervised contrastive learning does not require additional manual annotation.
To tackle these issues, we present a framework that leverages numerical evidence and moment contrast-based supervised contrastive learning for confusing LJP. Firstly, the framework proposes to extract numerical evidence (the total crime amount) from the fact description nition (NER) task for predicting the term of the penalty, where the recognized numbers make up the total crime amount. This formulation is able ad- dress the difficulty of directly deducing the precise total crime amount from the fact description where the numbers are scattered randomly throughout and only some of them are part of the crime amount, while others are not. Then, the extracted numer- ical evidence is infused into the term of penalty prediction model while preserving its numeracy, which is achieved by a pre-trained number encoder model (Sundararaman et al., 2020).
named entity recog-
We conduct extensive experiments on two real- world datasets (i.e., CAIL-Small and CAIL-Big). The experimental results demonstrate that the pro- posed framework achieves new state-of-the-art re- sults, obtaining up to a 1.6 F1 score improvement for confusing legal cases and a 3.73 Fl score im- provement for numerically sensitive legal cases. Ablation studies also demonstrate the effectiveness of cach component of the framework.
Secondly, in order to pull fact representations from the same class closer and push apart fact rep- resentations from confusing charges, the frame- work introduces the moment contrast- su- pervised contrastive learning(SCL; (Khosla et al.. 2020; Gunel et al., 2020; Suresh and Ong, 2021)) and explores the best strategy to construct positive example pairs to benefit all three subtasks of LJP simultaneously. The proposed moment contrast- based SCL addresses two challenges to applying the original in-batch SCL to LJP. The first chal- lenge is that the number of charge classes is signifi- cantly greater than the number studied in previous studies (Gunel et al., 2020) (e.g., 119 classes for charge prediction), which increases the difficulty of finding sufficient negative examples in the mini- batches. To address it, we introduce momentum update queue with large size for SCL, which al- lows for providing sufficient negative examples. The second challenge is when applying the orig- inal single-task SCL to the multi-task LJP there
Related Work
2.1 Legal Judgment Prediction
In recent years, with the increasing availability of public benchmark datasets (Xiao et al., 2018a; Feng et al., 2022b) and the development of deep learning, LJP has become one of the hottest topics in legal artificial intelligence (Yang et al., 2019; Zhong et al., 2020: Gan et al., 2021b; Cui et al., 2022; Feng et al., 2022a; Lyu et al., 2022). Our work focuses on confusing legal judgment predic- tion, which is a typical difficulty in LJP. To solve this challenge, Hu et al. (2018) manually annotates discriminative attributes for legal cases and gener- ates attribute- aware representations for confusing charges by attention mechanism. LADAN (Xu et al., 2020) extracts distinguishable features for law articles by removing similar features between nodes through graph neural network-based opera- tor. NeurJudge (Yue et al., 2021) utilizes the results of intermediate subtasks to separate the fact de-

---

down the following multi-task learning frame- work as shown in Figure 2. Specifically, firstly, a shared fact encoder used to encode f' into basic legal document representations.
Fact Description Text
Shared FatreedFattunmoderr
ChargeEncuderr
Encoder
TermEncoddr
H SharedFac(Ennoder((F)
(1)
The SharedFactEncoder could be Long-Short Term Memory Network (LSTM; (Hochreiter and Schmidhuber, 1997))) or pre-trained langugae mod- els, e.g.. BERT (Devlin et al., 2019).
Charge Prediction
Law Asticle Prediction
Term of Penalty Prediction
Figure 2: The multi-task learning framework of legal judgment prediction.
Secondly, in order to learn task-specific repre- sentations for each subtask (Zhong et al., 2018; Yue et al., 2021), different private encoders are built upon the basic shared encoder (Zhong et al., 2018; Yue et al., 2021). Specifically, we denote Charge LawEncoder and Termercoder as corresponding private encoders for the three sub- tasks as follows.
scription into different circumstances and exploits them to make the predictions of other subtasks.
2.2 Contrastive Learning
The purpose of contrastive learning (Chopra et al.. 2005) is to make similar examples closer together and dissimilar examples further apart in the fea- ture space. Contrastive learning has been widely explored for surf--uuerriiiduuuuprrisssseee represen- tation learning (Wu et al., 2018; Hjelm et al., 2018;
He ChargeLincoder((H) (H,)
(2)
Hi LawEncoder(HLy)
(3)
H, TermEncoder(HH)
(4)
Bachman et al., 2019; Chen et al., 2020; He et al.. 2020; Nguyen and Luu, 2021; Wu et al., 2022). Recently, several studies have extended contrastive learning to supervised settings (Gunel al., 2020; Khosla al., 2020; Suresh and Ong, 2021: Zhang al., 2022; Nguyen et al., 2022), where examples belonging to the same label in the mini-batch are re- garded as positive examples to compute additional contrastive losses. In contrast to previous studies, we present a framework that leverages MoCo-based supervised contrastive learning and numerical ev- idence, which is neglected by earlier studies for confusing LJP.
Thirdly, based on these task-specific represen- tations, different classification heads (e.g., multi- layer perceptron) and cross-entropy classification loss are used to compute the losses (i.e., le, &, 4,) for the three tasks. The training objective the sum of each task's loss as follows:
Lee =
(5)
Methodology
4.1 Overview
Fig 3(a) provides an overview of the proposed framework. Given a fact description f. on one hand, well-trained BERT-CRF- tity recognition model is used to extract the total crime amount from ff as numerical evidence, which is then encoded into representations for predicting the term of the penalty of f. On the other hand, Moco-based supervised contrastive learning for LJP and two strategies for constructing positive example pairs are introduced to compute the con- trastive loss. The final training loss is the weighted sum of the contrastive loss and three standard cross- entropy classification losses of subtasks.
3 Background
named en-
In this section, we formalize the LJP task and its
multi-task learning framework.
3.1 Problem Formulation
Let f = {81, (2,,82,..., s2, sN) denote the fact description of case, where sentence s; {w1, W2...., ..., WM} contains M words, and N is the number of sen- tences. Given fact description f, the LJP task aims at predicting its charge Ye C, applicable law article yn E and term of penalty yt E T.
4.2 Numerical Evidence for Term of Penalty Prediction
3.2 Multi-Task Learning Framework of LJJP
While previous studies have designed various neu- ral architectures for LJP, these models can be boiled
Numerical Evidence Extraction as NER. In the LJP datasets, there is no explicit total crime amount

---

Momentum Enceder
Similarity
Total Loss
a
Loss
Momeenum update
Sinsagy
Stutzgy 15
Charge Encoder
H
Charge Label
Classifer
H
Duserption
Fanc
I law
Hi
Law Classifer
Cross Entmpy
Taw Arricle
Label
Hl 11 Im
Pensity Term Label
Encoder
Classiftr
numerical
BERT-CRF
NumEncoder
Cress Furmpy
0-1 Package
fBLO
BERT-CRF
Numoeceder
Limon
(c)
CARUNER Dabeet
Figure 3: (a) Overview the proposed framework. (b) Illustration of the training process the numerical evidence extraction model. (c) Pre-training of the number encoder.
provided for each instance. To address this, we pro- pose to formalize the calculation of the total crime amount as Named Entity Recognition (NER) task, where the scattered numbers in the fact description that are part of the total crime amount will be rec- ognized named entities. Then, the of the recognized numbers is regarded as the final crime amount. The reason for this formalization is that recognizing which numbers are part of the crime amount is easier than directly computing the total crime amount from the fact description. Fig 3(b) illustrates the training process.
Algorithm 1: The 0-1 knapsack algorithm used for automatically constructing the CAE-NER dataset.
Input
:Fact description f {(s,, m,)h,,,,iirnm crime
amount T
Output SSlected sentenees E
Function BinarySelect ID:
/* Select a set of sentences, the sum of their crime amount is T
*/
2
for i 4 to do
3
if m: < then if BriiarySSelect (f. m,,jj j + then
E E=Eu(s} return True end end
Specifically, train the numerical extraction model on the dataset used for the Crime Amount Extraction (CAE) task2. To train the NER model, given an instance (f, T) in CAE, where f and T denote fact description and crime amount, respec- tively, we need to convert each instance into the NER format. To reduce expensive manual annota- tion costs, we propose 0-1 knapsack algorithm to automatically label named entities in f. The 0) knapsack algorithm finds set of sentences from where the sum of their numbers equals the crime amount Then the numbers in the selected sen- tences are labeled as named entities, Algorithm illustrate this construction process. Figure shows an example of converting an instance in CAE into the NER format. The converted dataset is named CAE-NER. based on which train the state-of- the-art BERT-CRE NER model, referred as the numerical evidence extraction model.
else
if m, = then
10
E= B (s}) return True
end end return 18
end
16 E BinarySelect(,.T.O0
return E
pseudo crime amount label annotated by the well-trained numerical evidence extraction model, denoted five-tuple (f, yes Ht, St. m).
Numerical Evidence Encoder. Given the ex- tracted numerical evidence, need numerical evidence encoder (NumEncoder) that should be pable of encoding the numerical evidence into hid- den representations while preserving numerical significance. To achieve this, we propose to pre- train NumEncoder with the following principle: the
Now, each instance the LJP dataset can obtain
/ www.tr.cuutroe.eeeeeeeee.eeee.e. court.gov.com wwwrrt.ov.ccn/ppeeeeee wtt.cc//aages/aac222111111 html

---

cosine similarity of the learned representations of a pair of numbers should have a linear relationship with respect to their numerical distance. Fig 3(c) illustrates the training process of NumEncoder.
the supervised contrastive loss as follows:
explq: kp/t)
lsup Σ
Σ
log
IEI P())
Σ expla: ka
PEP(
acA(1)
(11)
Specifically, given an automatically generated training data Dmum = ((x,, y/ )) where (ar; yi) represents a pair of numbers, we use the following training objective fnam to optimize the parameters of the LSTM-based NumEncoder:
where P(i) (tlyt ye E G and A(i)
{tje E L}. 4i is the query feature encoded by a query encoder fg(:;0)) 0g). kp- ka in Q are the key features encoded by a key encoder fie(;; 0s) Og are smoothly updated as follows:
X; NumEncoder((z))
(6)
0g t md + (1 mm
(12)
Y; NumEncoderrys)
(7)
where m is the momentum coefficient. Samples in Q and L are progressively replaced by the current mini-batch following a the ablation section, this MoCo-based SCL shows ad advantages over the standard in-batch SCL.
2 2| M/ |
mxwm
cos (xi, vs)
(8)
Vil
strategy.
where cos represents the cosine distance function.
Next, we explore two strategies to construct pos- itive example pairs to address the multi-task learn- ing challenge of LJP.
Infusing Numerical Evidence for Predicting the Term of Penalty. Lastly, we infuse the represen- tations of the numerical evidence into the term of the penalty prediction model. Specifically, given a training instance (f: Ye. Vt, Vt, m), its numerical evidence m encoded by the pre-trained number encoder and then is fused into the term of penalty prediction head as follows:
Strategy A straightforward strategy to com- pute a contrastive loss for each subtask of LJP, and then sum them into one loss. Formally, three fea- ture queues, i.e., Q Q' and Q', are used to store task-specific feature, i.e., He. H₁ and H.. Three label queues, i.e., L L and L are used to store subtask labels, i.e.. Me y and yt- The overall con- trastive loss is defined as follows:
Hm NumEncoder((m)
& = CrossFnntopyyMLLP(HH Hm.)) (10)
where [:] denotes the concatenation operation.
let = cf sup + 3e 0e sup
(13)
where Copp- Coup and et for each subtask computing by Eq. 11. The final training objective of Strategy I is defined by:
suuu are contrastive losses
4.3 MoCo-based Supervised Contrastive Learning for Confusing Judgment Prediction
foe + lat
(14)
Strategy II. When closely examining Eq. 13, we can observe the Contradictory Phenomenon as dis- cussed in Sec. 1. In Strategy I, glusk treats instances with the same subtask labels (e.g.. charge labels) as positive examples. However, these instances may have different other subtasks labels (e.g., applicalle law labels or term of penalty labels). As a result, ssu will force the SharedF act Encoder to learn features that benefit one subtask but degrade the performance of the other two tasks.
To address the challenges of large class numbers and the multi-task learning nature of LJP when ap- plying the original in-batch SCL, we introduce the momentum contrast (MoCo) (He et al., 2020) based SCL. Furthermore, we explore the best way to con- struet positive examples so that they can benefit all three subtasks of LJP simultaneously.
Firstly, we propose to augment the standard in- batch SCL with large-sized momentum update queue (He et al., 2020), allowing for providing suf- ficient samples for computing the contrastive loss. Specifically, we maintain one feature queue Q and one label queue L to store sample features and cor- responding labels. For each example <c;, 4> in the mini-batch Z, we select positive and negative sam- ples from @ based on the labels in L to compute
To solve this problem, we propose to view the in- stances whose three subtask labels are all the same as positive examples and impose the MoCo-based SCL on the shared features Hy. Specifically, we use a feature queue Q store the shared fea- tures Hy, and three label queues L C and L to store three subtask labels. Then the positive

---

Dataset
CAlL-Small
CAig.-Big
CAE
Methods
Charges Fl
Articles F1
Term
#Training Set Cases
101.619
1,588,381
3275
LADAN
82.42
75.87
34,28
#Validation Set Cases
13,769
13,769
273
LADAN+1
82.77
76.40
34.43
#Test Set Cases
26,749
185,290
500
LADAN+1I
83.83
77.04
35.71
#Charges
119
134
LADAN+1+II
83.47
77.38
35.77
#Law Articles
103
121
#Term of Penalty
11
11
Table 2: Effects of different supervised contrastive learn- ing strategies.
Table 1: Statistics of the used legal judgment predic-
tion datasets (CAlL-Small and CAIL-Big) and Crime Amount Extraction (CAE) dataset.
word embeddings provided by (Zhong al., 2018). More training details about the BERT-CRF NER model, the LJP model, and the NumberEncoder model can refer to Table the Appendix.
samples set for sample i is denoted as P(i) (g) u (q) = vi, ft (g) Dt q Q*) where Clack (q) denotes the label of index in each task label queue. Based on P(i), we use Eq. 11 compute the contrastive loss, denote as Sup Strategy II is able to address the Contradictory Phenomenon Strategy and improve the perfor- mance of all three subtasks.
5.3 Baselines
We compare our framework with the follow- ing state-of-the-art baseliens: HARNN (Yang et al., 2016), LADAN (Xu et al., 2020), Neur- Judge+ (Yue et al., 2021), CrimeBERT (Zhong et al., 2019). The details of these baselines are left in the Appendix.
The final training objective of Strategy II is de- fined by: = Loe + Seoup
(15)
5.4 Development Experiments
where À is hyperparameter.
To empirically evaluate which strategy is better for performing SCL. for multi-task LJP, we conduct development experiments on CAIL-small using the LADAN backbone. The results are listed in Table 2. As can be observed, firstly, both Strategy and II lead to improvements in LADAN's performance across the three subtasks. However, the gains of Strategy is much smaller than those of Strategy II, which verifies the existence of Contradictory Phenomenon in Strategy I. Furthermore, we also explore the effect of combining these two strategies. i.c., using la + lB as the supervised contrastive loss. As seen, the improvement of this combination is not significant.
5 Experiments
5.1
Datasets
To evaluate the effectiveness of our framework, we conduct experiments two real-world datasets (i.e., CAIL-Small and CAIL-Big) (Xiao et al., 2018b). Each instance in both datasets contains one fact description, one applicable law article, one charge, and one term of penalty. To ensure a fair comparison, we use the code released by (Xu et al., 2020) to process the data. All models are trained on the same dataset. The crime Amount Extraction (CAE) dataset is also real-world dataset from the Chinese Legal Al challenge 3. Table shows the statistics of the used datasets.
Consequently, the remaining experiments, we adopt Strategy II as the contrastive loss, unless otherwise specified. The final method combined the MoCo-based SCL and numerical evidence is denoted as NumSCL.
To specifically evaluate our framework on con- fusing legal cases, we define set of confusing and Lumber-sensitive charges. Due to page limitations, the details and statistics of these charges definitions are listed in Table 9 in the Appendix
5.5 Main Results
To evaluate the effectiveness of the proposed frame- work, we augment each baseline with NumSCL and conduct experiments the CAIL-Small and CAIL-Big datasets. Due the expensive training cost and the large size of the training dataset, we did not evaluate CrimeBERT on CAIL-Big follow- ing (Yue et al., 2021). The results are listed Table and Table 4.
5.2 Implementation Details
We follow (Xu et al., 2020) to conduct data pre- processing. The THULAC tool is used to segment Chinese into words. The word embedding layer in the neural network is initialized by pre-train
'http://data..r court. wowrt.gov..oo. www.rr/gw..cc/pgge//laaaaa// html htttts:///ithub.. com/thunnp/TTHLAC

---

Tasks
Charges
Law Articles
Term Penalty
Metrics
Acc.
MP
MR
F1
Acc.
MP
MR
F1
Acc.
MP
MR
F1
HARNN
84,54
82.56
82.94
82.26
80.09
76.46
77.69
75.95
38.38
36.12
33.99
34.32
W/NumSCI.
85.26
83,93
83.76
83.39
81.07
77.95
78.52
77.11
39.18
37.32
34.50
35.03
LADANWT.
84,90
82,35
83,26
82.42
80.38
75.84
77.84
75.67
38.21
35.95
34.01
34.28
w/NumSCI.
85.37
83.91
84.04
83.57
81.32
78.06
78.59
77.24
39.38
37.95
35.23
35.95
NeurJudge* +
83,25
82.11
81.69
81.3
80.95
77.93
78.59
77.00
37.88
37.20
33.82
34.92
w/NumSCL
84.45
83.30
83.55
82.88
81.12
78.10
78.98
77.32
39.65
39.48
34,65
36.22
CrimeBERT
86.61
85.04
84.72
84.51
82.33
79.38
79.72
78.46
39.34
38.66
35.48
36.58
w/NumSCL
85,91
85.71
85.98
85.54
82.63
80.10
80.88
79.50
39.72
38.50
35.84
36.67
Table Main results CAIL--mall. Acc., MP, and MR short for accuracy, macro precision, and macro recall, respectively.
Tasks
Charges
Law Artieles
Term of Penalty
Metries
Acc.
MP
MR
F1
Acc.
MP
MR
F1
Acc.
MP
MR
F1
HARNN
96.48
88.10
83.54
85.34
96.54
84.88
79.42
81.40
60.30
51.08
47.47
48.79
w/NumSCL
96.55
88.84
83.96
85,82
96.60
86.63
81.16
82,99
60.20
52.18
47.50
49.06
LADANWT.
96.56
88.58
84.17
85,95
96.61
86.40
80.46
82,52
60.44
51.56
48.67
49.78
w/NumSCI.
96.65
89.34
84.57
86.49
96.71
87.60
81.74
83.68
60,56
51.85
48,86
49.81
NeurJudge" ?
95.48
85.57
79.55
81.49
96.26
85.78
81.38
82.80
58.40
49.67
43.32
44.90
w/NumSCL
95.73
86.37
80.88
82.58
96.20
85.48
81.56
82.78
58.40
49.10
43.54
45.43
Table 4: Main results CAIL-Big, Acc., MP and MR are short for accuracy, macro precision, and macro recall, respectively.
From Table 3 and Table 4, we make the fol- lowing observations. Firstly, the proposed frame- work can improve all the baselines and achieve new state-of-the-art results on the two datasets. Specifi- cally, on CAIL-Small, the absolute improvements reach up to 1.15, 1.57, and 1.67 F1 scores for the charge, law article, and term of penalty predictions, respectively. Secondly, on CAIL.-Big the gains are smaller, giving absolute improvements of 0.54, 1.16, and 0.53 Fl scores for the charge, law arti- cle, and term of penalty predictions, respectively. Thirdly, we observe that CAIL.-Big, NeurJudge gives worse performance than the other baselines. We hypothesize that the complex neural network architecture of NeurJudge+ may lead to overfitting on the CAIL-Big dataset. Lastly, for CrimeBERT, our framework can still obtain an absolute improve- ment of 1.03 and 1.04 Fl scores for charge and law article predictions. The overall gain on the term of penalty prediction is slight, however, when specifi- cally evaluating number-sensitive legal cases, the improvement can still be up to a 1.79 Fl score, as shown in Table 5.
Models
Acc.
F1
Num. F1
HARNN
38.38
34.32
28.13
w/ SCL
38.62
34.45
28.24(t 0.11)
w/ NumSCL.
39.18
35.03
29.15(f 1.02)
LADAN
38.21
34.28
26.83
w/ SCL
39.50
35.71
28.54ct 1.71)
w/ NumSCL
39.38
35.95
30.56(1 3.73)
NeurJudge*
37,88
34.92
27.30
w/ SCL.
38.11
34.96
27.35(1 0.05)
w/ NumSCL
39.65
36.22
29.03(t 1.73)
CrimeBERT
39.34
36.58
28.57
w/ SCL
39.71
36.48
28.99(t 0.42)
w/ NumSCL. 39.68
36.66
30.36(1 1.79)
Table 5: Effects of the proposed method on the term of penalty prediction of number- senberrsensitive legal cases. Num. Fl is the Fl score on the defined number-sensitive charges.
shown in Table 5, the improvement of MoCo- based SCL only is relatively tiny, only giving 0. and 0.05 F1 score improvements for HARNN and NeurJudge on number-sensitive legal cases. How- ever, when the models are further provided with the extracted numerical evidence, the Fl scores of all the baselines have a considerable boost. In particular, LADANMIL obtains a 3.73 Fl score improvement on number-sensitive cases. These re- sults show that the extracted crime amount is more beneficial for number-sensitive legal cases.
5.6 Ablation Studies
Effect of Numerical Evidence for Number- Sensitive Legal Cases. To examine the effect of numerical evidence for predicting the term of penalty, we conduct ablative experiments. As
Effect of Contrastive Learning for Confusing Charges. We conduct experiments to validate the

---

Models
Ace.
F1
Conf. FI
Example 1
HARNN
84,54
82.26
75,60
Fact Descriptions: XXX, dofendant XXX lought with victim XXX due to trivial matter in a market in XXX Town. XXX and wanted take revenge the victm XXX. At XXX of the same year, the defendant XXX guthered three people including XXX, XXX and XXX. and fled the open parking lot next to a market. The delendant XXX shashed the vietim XXX's right with watermelon knile, then they fled scone, According to forensic identification, victim XXX suffored minor injarics
w/ NumSCL
85.26
83.39
77.19 (1 1,59)
LADAN
84,90
82.42
75.29
w/ NumSCI.
85.37
83.57
76.76 (1 1.47)
NeurJudge*
83.25
81.30
73.42
w/ NumSCL
84,45
82.88
75.97()2.55)
Table 6: Effects of the proposed method the charge prediction of confusing legal cases. Conf. Fl is the Fl score on the defined confusing charges.
Golden Charge: Crime Picking Quiiell
Goden Law Article: 34, Golden Term of Penalty: 6
LADANsm Crime Intention Injury; Law Article Term
Penally:
AiDANur-NumSSC Crime of Picking Quarcel: Law Article 34: Term Penalty:
Models
Charge Fl Law
Term F1
Example 2
LADAN
82,42
75.67
34,28
LADANsCL
83,
76.24
36,12
Fact Descriptions: From XXX to XXX. the defendant XXX committed three thets XXX City. 1, At X XXX, 2015, the defendant XXX stole "Dayang, brand" electric trieyele worth $5,600 from the victim XXX. 2. On XXX, 2015, the defendant XXX "Dayang" electric tricycle wort $4,800 the
LADANKinSe..
83,57
77.24
35,95
Table 7: Effect of momentum contrast queue.
victim XXX in XXX city. 3. At XXX. 2016, the defendant XXX electric vehicle battery from vietim discovered and altempted.
effect of contrastive learning for predicting charges, particularly confusing charges. As shown in Ta- ble 6, the absolute FI score improvements of con- fusing charges are greater than those of the overall charges. For example, NeurJudget obtains an ab- solute 2.55 Fl score improvement on confusing charges, which demonstrates the effectiveness of the moco-based SCL in learning distinguishable representations for confusing charges.
Golden Charge: Crime of Theft,
Golden Law Article 32; Golden Term of Penalty:
LADAN Crime of Theft: Law Article 56: Term of Penalty:9
LADAN LADAN.n-NumSSCI. Crime of Theft; Law Aniile 56; Term
Penally: 7; Extracted Crime Amount: $10,400
Figure 4: Qualitative examples to demonstrate the effect of the proposed framework.
02-
Effect of Momentum Contrast Queue. We uti- lize LADAN as the backbone for comparing Num- SCL with the in-batch SCL(SCL) which takes the current mini-batch the lookup dictionary to com- pute the contrastive loss. The in in-batch SCL is trained using the same parameters as the MoCo- based SCL. As depicted in Table 7, the in-batch SCL yields superior results when combined with LADAN but underperforms NumSCL in charge and law article prediction tasks. This highlights the advantage of employing large queue as the lookup dictionary in SCL. We also observe that the perfor- mance of the term of penalty is not significantly af- fected by the choice of the lookup dictionary. This observation aligns with the previous finding that the extracted numerical evidence more beneficial than SCL for the term of penalty prediction.
70-
Charge
74-
Article
Tem
38
36-
34
1.0
Lambda
Figure 5: The impact of À in Eq.15 for the three subtasks of L.P.
6 Case Studies
Figure 4 shows two cases to qualitatively demon- strate the effect of the proposed framework. In the first case, LADANMT. incorrectly predicts the case's 's charge into its confusing charge Crime of Intentional Injury, which should be Crime of Picking a Quarrel. However, with the proposed framework, this error is corrected. The second case demonstrates the effect of numerical evidence. LADANMIL incorrectly predicts the case's 's term of penalty as label 9, meaning a sentence of fewer than months. Given the accurately extracted crime amount of $10,400, which is a relatively large crime amount, the model correctly predicts
Effect of A. We carry out experiments to verify the impact of X in 15. As illustrated in Fig- ure5, with the increasing of X, the performance of charge and law article predictions correspondingly improves. We also observe fluctuation in the term of penalty prediction, again showing that the ex- tracted numerical evidence plays a more significant role than SCL for predicting the term of penalty.

---

the term of penalty label 7, meaning sentence of more than months but less than 12 months.
Acknowledgements
This work was supported in part by National Key Research and Development Program of China (2022FFC3340000, National Natural Science Foundation of China (62376243, 62037001), the Starry Science Fund of Zhejiang University Shanghai Institute for Advanced Study (SN-ZJU- SIAS-00100 Project by Shanghai AI Laboratory (P22KS00111) Program of Zhejiang Province Sci- ence and Technology (2022C01044) the Funda- mental Research Funds for the Central Universities (226-2022-00142, 220000442222222222222 226-2022-00051))
7 Conclusion
In this paper, we present framework that intro- duces MoCo-based supervised contrastive learning and weakly supervised numerical evidence for con- fusing legal judgment prediction. The framework is capable of automatically extracting numerical evidence for predicting number-sensitive cases and learning distinguishable representations to benefit all three subtasks of LJP simultancously. Extensive experiments validate the effect of the framework.
Limitations
References
While the used 0-1 knapsack algorithm enjoys the merit of automatically constructing training dataset for building the NER model, it cannot accu- rately calculate the crime amount when the suspects return some properties to the victims as the returned properties should be subtracted from the amount of the crime. More sophisticated techniques could be developed to calculate the amount of crime more precisely.
Nikolaos Aletras, Dimitrios Tsarapatsanis, Daniel Preotiuc-Pietro, and Vasileios Lampos. 2016. Pre- dieting judicial decisions of the european court of human rights: A natural language processing per- spective. PeerJ Computer Science, 2:e93.
Philipp Bachman, R Devon Hjelm, and William Buchwal- ter. 2019. Learning representations by maximizing mutual information across views. Advances neural information processing systems, 32.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Ale- tras. 2019. Neural legal judgment prediction in glish. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4317-4323.
Our LJP research focuses on Chinese legal docu- ments under the jurisdiction of the People's ' Repub- lic of China. While the framework was developed and tested specifically for the 3-task Chinese Legal Judgment Prediction (LJP), we believe the under- lying methodology could be generalized and ap- plied to other LJP tasks, even those from different jurisdictions. However, this would likely require modifications account for the unique character- istics and complexities of each jurisdiction's 's legal system. We will leave this for future work.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. ternational conference on machine learning, pages 1597-1607. PMLR.
Sumit Chopra, Raia Hadsell, and Yann LeCun. 2005. Learning similarity metric discriminatively, with application to face verification. In 2005 IEEE Com- puter Society Conference on Computer Vision and Pattern Recognition (CVPR 2005), 20-26 June 2005, San Diego CA, USA, pages 539-546. IEEE Com- puter Society.
Ethical Concerns
Due to the sensitive nature of the legal domain, applying artificial intelligence technology to the legal field should be carefully treated. order to alleviate ethical concerns, we undertake the follow- ing initiatives. First, to prevent the risk of leaking personal private information from the evaluated real-world datasets, sensitive information, such names of individuals and locations, has been anonymized. Second, suggest the predictions generated by our model should serve supportive references to assist judges making judgments more efficiently, rather than solely determining the judgments.
Junyun Cui, Xiaoyu Shen, Feiping Nie, Zheng Wang. Jinglong Wang, and Yulong Chen. 2022. A survey legal judgment prediction: Datasets, metries, models and challenges. arXiv preprint arXiv:2204.04853
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. 2019, Bert: Pre-training of deep bidirectional transformers for language understand- ing. Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies, Volume (Long and Short Papers), pages 4171- 4186.

---

Yi Feng, Chuanyi Li, and Vincent Ng. 2022a. Legal judgment prediction: A survey of the state of the art. In Proceedings of the Thirty-First International Joint Conference on Artificial Intelligence, IJCAI-22, pages 5461-5469. International Joint Conferences on Artificial Intelligence Organization. Survey Track.
Yougang Lyu, Zihan Wang, Zhaochun Ren, Pengjie Ren. Zhumin Chen, Xiaozhong Liu, Yujun Li, Hongsong Li, and Hongye Song. 2022. Improving legal judg- ment prediction through reinforced criminal element extraction. Information Processing & Management, 59( 102780.
Yi Feng. Chuanyi Li, and Vincent Ng. 2022b. Legal judgment prediction via event extraction with con- straints. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 648 664.
Thong Nguyen and Anh Tuan Luu. 2021. Contrastive learning for neural topic model. Advances in neural information processing systems, 11111994-1988
Thong Nguyen, Xiaobao Wu, Anh Tuan Luu, Zhen Hai, and Lidong Bing. 2022. Adaptive contrastive learn- ing on multimodal transformer for review helpfulness prediction. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 10085-10096.
Leilei Gan, Kun Kuang, Yi Yang, and Fei Wu. 202 la. Judgment prediction via injecting legal knowledge into neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11866-18874,
Dhanasekar Sundararaman, Shijing Si, Vivek Subra- manian, Guoyin Wang. Devamanyu Hazarika, and Lawrence Carin. 2020. Methods for numeracy- preserving word embeddings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, Novem- ber 16-20, 2020, pages 1742-4753 Association for Computational Linguistics.
Leilei Gan, Yating Zhang. Kun Kuang, Lin Yuan, Shuo Li, Changlong Sun, Xiaozhong Liu, and Fei Wu. 2021b. Dialogue inspectional summarization with factual inconsistency awareness. arXiv preprint arXiv:2111.03284.
Beliz Gunel, Jingfei Du, Alexis Comneau, and Veselin Stoyanov. 2020. Supervised contrastive learning for pre-trained language model fine-tuning, In Interna- tional Conference on Learning Representations.
Varsha Suresh and Desmond Ong. 2021. Not all neg- atives are equal: Label-aware contrastive loss for fine-grained text classification. In Proceedings of the 202 Conference on Empirical Methods in Natural Language Processing, pages 4381 18344
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. 2020. Momentum contrast for unsu- pervised visual representation learning. In Proceed- ings of the IEEE/CVF conference computer vision and pattern recognition, pages 9729-9738.
Xiaobao Wu, Anh Tuan Luu, and Xinshuai Dong. 2022. Mitigating data sparsity for short text topic modeling by topic-semantic contrastive learning. In Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2748-271 2760.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2018. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 0868.06670.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. 2018. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference computer vision and pat- tern recognition, pages 3733-3742.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 173):1735- 1780.
Zikun HU, Xiang Li. Cunchao Tu, Zhiyuan Liu, and Maosong Sun. 2018 Few-shot charge prediction with discriminative legal attributes. In Proceedings of the 27th International Conference on Computational Linguistics, pages 487-498.
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo. Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, and Jianfeng Xu. 2018a. CAIL2018: A large-scale legal dataset for judgment prediction. CoRR, 00l/1807.024777
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. 2020. Su- pervised contrastive learning. Advances in Neural Information Processing Systems. 33:18661-18673.
Chaojun Xiao, Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Zhiyuan Liu, Maosong Sun, Yansong Feng, Xianpei Han, Zhen Hu, Heng Wang, et al. 2018b. Cail2018: 88: A large-scale legal dataset for judgment prediction. arXiv preprint arXiv: 02478.
Yifei Liu, Yiquan Wu, Yating Zhang, Changlong Sun, Weiming Lu, Fei Wu. and Kun Kuang. 2023, MI- lip: Multi-law aware legal judgment prediction. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 1023-1034.
Nuo Xu, Pinghui Wang. Long Chen, Li Pan. Xiaoyan Wang, and Junzhou Zhao. 2020. Distinguish con- fusing law articles for legal judgment prediction. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguisticx, pages 3086- 3095.

---

Wenmian Yang, Weijia Jia, Xiaojie Zhou, and Yutao Luo. 2019. Legal judgment prediction via multi- perspective bi-feedback network. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 4085-4091.
Model
Parameter
Value
Word embedding size
200
Maximum document length
512
Maximum sentence num
15
Batch size
128
Learning rate
0.001
Zichao Yang. Diyi Yang. Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 1480-1489, San Diego, California. Associa- tion for Computational Linguistics.
LIP
Training epoch
16
Optimizer
Adam
Momentum queue Q size
65536
Momentum coefficient
0.999
Temperature a. A.
0.07
2, 2.
Model
BERT-CRF
Batch size
16
NER
Learning rate
0.00001
Training epoch
20
Linan Yue, Qi Liu, Binbin Jin, Han Wu, Kai Zhang, Yanqing An, Mingyue Cheng. Biao Yin, and Day- ong Wu. 2021. Neurjudge: a circumstance-awan neural framework for legal judgment prediction. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Infor- mation Retrieval, pages 973-982.
Optimizer
Adam
Training data size
128000
Max number, Min number
300000
Model
GRU
Word embedding size
200
NumEncoder
Hidden state size
256
Batch size
128
Learning rate
0.001
Han Zhang, Zhicheng Dou, Yutao Zhu, and Ji-Rong Wen. 2023. Contrastive learning for legal judgment prediction. ACM Transactions on Information Sys- tems, 4 144:11225
Training epoch
100
Optimizer
Adam
Table 8: Hyper parameter values
Shu Zhang, Ran Xu, Caiming Xiong, and Chetan Ra- maiah. 2022. Use all the labels: A hierarchical multi- label contrastive learning framework. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pages 16660-16668
the fact description into different representations and exploits them to make the predictions of other subtasks; (4) CrimeBERT (Zhong et al., 2019): CrimeBERT is initialized by BERT (Devlin et al., 2019), then is further pre-trained on crime data, giving better results than BERT. It is worth noting that the proposed method is model-agnostic which can be used to improve any other LJP models.
Haoxi Zhong, Zhipeng Guo, Cunchao Tu, Chaojun Xiao, Zhiyuan Liu, and Maosong Sun. 2018. Legal judg- ment prediction via topological learning. In Proceed- ings of the 2018 Conference on Empiricul Methods in Natural Language Processing, pages 3540-3549.
Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang. Zhiyuan Liu, and Maosong Sun. 2020. How does nlp benefit legal system: summary of legal artificial intelligence. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5218 -2300.
B Implemantation Details
For training the BERT-CRE NER model, we use the Adam optimizer and set the learning rate to 1e-5. The batch size is set to 16. We train the model for 20 epochs and select the best model on the validation set for testing. The best model on the test set of CAE can achieve a competitive accuracy of 0.875.
Haoxi Zhong, Zhengyan Zhang, Zhiyuan Liu, and Maosong Sun. 2019. Open chinese language pre- trained model zoo. Technical report.
A Baselines
For pre-training the numerical evidence encoder model NumEncoder, we synthesize a training dataset of size 128,000 where each pair of numbers (I,, Y:) are uniformly sampled from [0, 30, 0000]. A GRU model with a hidden size of 256 is used to model the number sequence. Adam is used to optimizing the parameters, and the learning rate is le-3. We train the NumEncoder model for 100 epochs.
We compare our method with the following non-pretrained and pre-trained models: (1) HARNN (Yang et al, 2016): an RNN-based neural network with a hierarchical attention mechanism for document classification; (2) LADAN (Xu et al., 2020): LADAN distinguishes confusing law ar- ticles by extracting distinguishable features from similar law articles using a graph-based method; (3) NeugeJudge+ (Yue et al., 2021 N:NurJudge++ uti- lizes the results of intermediate subtasks to separate
For training the LJP model, we use the Adam optimizer and set the learning rate to le-3. The

---

Fact Description:
The court found that at XXX, the defendant XXX stole black rectangular wallet and white cell phone (Meitu M4 brand, worth $900MONEV) from the victim XXX, while he was asleep. The wallet contained cash of S2600MONEY, After Being arrested, the defendant retumed the aforementioned cell phone and cash $2,000 the victim.
Crime Aoontt $3,500
Figure An example of converting an instance in the CAE dataset into the sample for training the NER model.
% Training Set
Charge Type
#Classes
size
Confusing Charges
41
37.13%
Number-sensitive Charges
33
22.25%
Table 9: Statistics of the defined confusing charges and number-sensitive charges.
batch size is set to 128. We train the model for 16 epochs and select the best model on the validation set for testing. In contrastive learning, the MoCo queue size and the temperate / are set to 65536 and 0.07, respectively. We run each experiment with five different seeds and report the averaged results. Table shows the detailed statistics of the used datasets.
C Evidence Extraction
Figure 6 shows an example of converting an in- stance CAE into the NER format.
D Definition of Confusing Charges.
To specifically evaluate method confus- ing legal cases, define confusing charges us- ing the predicted results of the baseline model LADANMIL. Concretely, the number that the
model incorrectly classifies class A into class B exceeds the pre-defined threshold, classes A and B will be added to the confusing classes. The defini- tion of the number-sensitive charges is determined by an experienced legal expert. The statisties of the definition of confusing charges and number- sensitive charges are listed in Table 9.


Reviews:
[{"review_id": "mImGZSFHlN", "reviewer": "Reviewer_zrQE", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: There are no confidence intervals reported for the results in Tables 3, 4, 5, 6, and 7. \n\nA1: Thank you for your valuable comments. For the CAIL-Small dataset, as indicated in Lines 763 - 764, I would like to clarify that five different random seeds (10, 512, 1024, 2020, 9090) were used to initialize the weights of the models and shuffle the input data to ensure the robustness and reliability of the experimental results. Therefore, the results in Table 3 are averaged results. The results with confidence intervals of Table 3 are reported as follows:\n\n| Metrics    | Charge Acc | Charge MP | Charge MR | Charge F1 | Law Acc | Law MP | Law MR | Law F1 | Term Acc | Term MP | Term MR | Term F1 |\n|-----------|---------|---------|---------|---------|---------|---------|---------|---------|---------|----------|----------|----------|\n| HARNN     | 84.54±0.56 | 82.56±0.14 | 82.94±0.19 | 82.26±0.05 | 80.09±2.63 | 76.46±0.59 | 77.69±0.26 | 75.95±0.52 | 38.38±0.47 | 36.12±0.91 | 33.99±1.75 | 34.32±0.50 |\n| w/NumSCL  | 85.26±0.14 | 83.93±0.12 | 83.76±0.10 | 83.39±0.09 | 81.07±1.35 | 77.95±1.48 | 78.52±0.69 | 77.11±1.42 | 39.18±0.20 | 37.32±0.61 | 34.50±0.85 | 35.03±0.14 |\n| LADAN_MTL | 84.9±0.42  | 82.55±0.24 | 83.26±0.13 | 82.42±0.20 | 80.38±0.16 | 75.84±0.32 | 77.84±0.24 | 75.67±0.27 | 38.21±0.17 | 35.95±0.81 | 34.01±0.78 | 34.28±0.03 |\n| w/NumSCL  | 85.37±0.21 | 83.91±0.27 | 84.04±0.13 | 83.57±0.20 | 81.32±0.01 | 78.06±0.48 | 78.59±0.21 | 77.24±0.26 | 39.38±0.21 | 37.95±1.15 | 35.23±0.62 | 35.95±0.29 |\n| NeurJudge+| 83.25±0.32 | 82.11±0.09 | 81.69±0.41 | 81.30±0.17 | 80.95±0.11 | 77.93±0.34 | 78.59±0.55 | 77.00±0.46 | 37.88±0.66 | 37.20±0.56 | 33.82±0.41 | 34.92±0.38 |\n| w/NumSCL  | 84.45±0.39 | 83.3±0.29  | 83.55±0.17 | 82.88±0.13 | 81.12±0.68 | 78.10±0.79 | 78.98±0.29 | 77.32±0.47 | 39.65±1.09 | 39.48±0.52 | 34.65±1.04 | 36.22±1.07 |\n| CrimeBERT | 86.61±0.10 | 85.04±0.27 | 84.72±0.08 | 84.51±0.17 | 82.33±0.14 | 79.38±0.72 | 79.72±0.20 | 78.46±0.36 | 39.34±0.42 | 38.66±1.02 | 35.48±0.14 | 36.58±0.22 |\n| w/NumSCL  | 86.91±0.10 | 85.71±0.05 | 85.98±0.06 | 85.54±0.01 | 82.63±1.40 | 80.10±0.09 | 80.88±0.11 | 79.50±0.16 | 39.72±0.19 | 38.50±0.50 | 35.84±0.14 | 36.67±0.23 |\n\nTable 3: Main results on CAIL-Small. Acc., MP, and MR are short for accuracy, macro precision, and macro recall, respectively. We run the experiments with five different seeds and report the averaged results and confidence intervals.\n\nFor the CAIL-Big dataset, in our preliminary experiment, we found that due to the large dataset size, experimental results are less affected by the random seed. Therefore, we randomly choose a seed and use this seed for all the experiments.\n\nWe will update these tables with confidence intervals in the revised version.\n\nQ2: The momentum contrast (MoCo) is not explained comprehensively. It is very difficult to find its concrete definition in the paper.\n\nA2: Thank you for your valuable comments. Here we illustrate the MoCo-based supervised contrastive learning again. \n\nWe propose to augment the standard single-task supervised contrastive learning with a momentum update queue~\\cite{he2020momentum}. The queue size can be much larger than a typical mini-batch size, thus can provide sufficient samples for computing the contrastive loss. Specifically, we maintain one feature queue $\\mathcal{Q}$ to store sample features of interest, and one label queue $\\mathcal{L}$ to store corresponding sample labels. Given $\\mathcal{Q}$ and $\\mathcal{L}$, for each example <$e_i$, $l_i$> in the mini-batch $\\mathcal{I}$, according to the labels in $\\mathcal{L}$, we can select positive and negative samples from $\\mathcal{Q}$ to compute the supervised contrastive loss using Eq. (11).\n\nIn Eq. (11), $q_i$ is the query feature encoded by a query encoder $f_q$ whose parameters are denoted as $\\theta_q$. The features $k_p$, $k_a$ in $\\mathcal{Q}$ are encoded by a momentum key encoder $f_k$ whose parameters are $\\theta_k$. $\\mathcal{Q}$, $\\mathcal{L}$, and $\\theta_k$ are updated as follows: \n$$\n\\theta_{k} \\leftarrow m\\theta_{k-1} + (1-m)\\theta_q\n$$\n\n$$\n\\text{Dequeue}(\\mathcal{Q}), \\text{Enqueue}(\\mathcal{Q}, k_i)\n$$\n\n$$\n\\text{Dequeue}(\\mathcal{L}), \\text{Enqueue}(\\mathcal{L}, l_i)\n$$\n\nwhere $m \\in [0, 1)$ is a momentum coefficient. $\\textit{Dequeue}$ and $\\textit{Enqueue}$ are operations to remove the element at the front of the queue and insert the element at the end of the queue, respectively. $k_i = f_k(x_i;\\theta_k)$. Note that $\\theta_k$ are fixed during the back-propagation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well-written and understandable.\nThe problem of confusing legal judgment prediction is very important and relevant to the conference.\nTheir novelty for extraction of numerical evidence as a named entity recognition task is very neat.", "reasons_to_reject": "Major:\nThere are no confidence intervals reported for the results in tables 3, 4, 5, 6, and 7. Must repeat the experiments for different random seeds.\nThe improvements in performance are mostly very small and it is not clear if those small improvements are statistically significant. \n\nMinor:\nThe momentum contrast (MoCo) is not explained comprehensively. It is very difficult to find its concrete definition in the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Legal Judgment Prediction (LJP) involves forecasting the charge, relevant law article, and penalty duration for a legal case based on its factual description. LJP encounters a central obstacle for differentiating the legal cases that display subtle variations in text or numeric values. \nTo authors propose a MoCo-based supervised contrastive learning and weakly supervised numerical cues for classifying LJP scenarios. \nThey suggest the novel approach of using named entity recognition task to enhance the identification of numerical cues. \nTheir proposed framework also incorporates MoCo-based supervised contrastive learning to facilitate multi-task LJP and investigates optimal strategies for forming positive example pairs that concurrently benefit all three LJP subtasks. \nThey try their proposed methods on real world data and compare their final framework with ablated frameworks and with baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IBLHSvYIpY", "reviewer": "Reviewer_kzgp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The paper lacks a comprehensive review and comparison with existing LJP methods [1,2, 3].\n\nA1: Thank you for your valuable comments and suggestions. Here we give a review and comparison of these three methods. \n\n[1] proposes a relational attention module to impose a relational constraint on the obtained <similar case, case, dissimilar case> to more effectively utilize the beneficial information contained in the data. Different from [1],  our framework leverages MoCo-based supervised contrastive learning and weakly supervised numerical evidence for confusing LJP. \n\n[2] proposes a supervised contrastive learning framework to distinguish various law articles within the same chapter of a Law and similar charges of the same law article or related law articles.  Our work has two key differences compared with [1]. Firstly, we use a moco-based supervised contrastive learning which can provide sufficient negative examples for computing contrastive loss in LJP. Secondly, in addition to supervised contrastive learning, we also propose to extract numerical evidence (the total crime amount) from the fact description for predicting the term of the penalty.  The effect of numerical evidence can be found in Table 5 of our work and Table 3 of [2], which shows that only using supervised contrastive learning does not significantly improve the performance of term of penalty prediction.  \n\n[3] proposes a reinforcement learning (RL) based framework to extract four types of criminal elements (i.e., the criminal, target, intentionality, and criminal behavior) to distinguish similar fact descriptions. Different from [3], our work proposes to extract numerical evidence (the total crime amount) from the fact description as a named entity recognition (NER) task for predicting the term of the penalty. \n\nWe will include and discuss these related papers in our revised version.\n\nQ2: The Supervised Contrastive Learning (SCL) methods proposed within this research did not demonstrate a significant enhancement in the ablation studies. This calls for further investigation and optimization.\n\nA2: The SCL method is more beneficial for confusing charge legal cases than number-sensitive legal cases. As shown in Table 6, the absolute F1 score improvements of confusing charges are greater than those of the overall charges. However, when conducting ablative experiments to examine the effect of numerical evidence for predicting the term of penalty, as shown in Table 5, the improvement of MoCo-based SCL only is relatively tiny.  When the models are further provided with the extracted numerical evidence, the F1 scores of all the baselines have a considerable boost. These results show that the extracted crime amount is more beneficial for number-sensitive legal cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed methodology capitalizes on the numerical data embedded within LJP texts, demonstrating a notable improvement in performance.\n\n2. The authors innovatively apply a momentum contrast-based supervised contrastive learning approach, addressing the issue of large class numbers and the subsequent difficulty in identifying sufficient negative examples within mini-batches.\n\n3. The introduced framework exhibits versatility, being adaptable to current encoder-based methodologies.", "reasons_to_reject": "1.\tThe paper lacks a comprehensive review and comparison with existing LJP methods [1,2, 3].\n2.\tThe Supervised Contrastive Learning (SCL) methods proposed within this research did not demonstrate a significant enhancement in the ablation studies. This calls for further investigation and optimization.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a novel framework, termed MoCo-based Supervised Contrastive Learning for Multi-task Legal Judgement Prediction (LJP), designed to alleviate the complexity inherent in deciphering legal cases. The approach incorporates numerical evidence extraction, addressing the challenge of predicting penalty terms based on dispersed numerical data. Empirical results indicate a performance enhancement over existing methodologies.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] Augmenting Legal Judgment Prediction with Contrastive Case Relations\n[2] Contrastive Learning for Legal Judgment Prediction\n[3] Improving legal judgment prediction through reinforced criminal element extraction"}}, {"review_id": "0H7bX9eOwx", "reviewer": "Reviewer_VZ2V", "summary": "", "questions": "- The authors mention that using standard cross-entropy is not an ideal solution, since \"The model should be punished more if it classifies a charge into its corresponding confusing charges.\". This sounds counter-intuitive to me since misclassifications naturally occur in closed-related subjects (close-related labels, e.g., similar types of crimes). I don't understand why such misclassifications should be penalized more compared to misclassifications in favor of very unrelated crimes. Intuitively, misclassifying the crime of \"Picking a Quarrel\" as \"Intentional Injury\" sounds less problematic compared to picking a much more unrelated crime such as \"Drug Trafficking\" for instance. Can you explain the reasoning (intuition) behind the proposed solution?", "limitations": "", "rating": 3, "confidence": 1, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Q1: The proposed framework is built around a specific setting (3-task Chinese LJP), and there is no evidence that a similar methodology would be useful in other related LJP tasks from other jurisdictions (Chalkidis et al., 2019; Niklaus et al., 2021; and others).\n\nA1: Thank you for bringing up this important point. While our framework was developed and tested specifically for the 3-task Chinese Legal Judgment Prediction (LJP), we believe the underlying methodology could be generalized and applied to other LJP tasks, even those from different jurisdictions. However, this would likely require modifications to account for the unique characteristics and complexities of each jurisdiction's legal system. We will leave this for future work.\n\nQ2: The authors mention that using standard cross-entropy is not an ideal solution, since \"The model should be punished more if it classifies a charge into its corresponding confusing charges.\". \n\nA2:  Thank you for your valuable comments. We found that standard cross-entropy classification loss struggles to distinguish confusing charges, therefore we propose an additional contrastive learning loss to punish the model more if it classifies a charge into its corresponding confusing charges. In other words, the total loss will increase if the model classifies a charge into its corresponding confusing charges. In this way, the model can learn to avoid such errors. This approach encourages the model to pay closer attention to the subtle differences between closely related charges, thereby improving its overall accuracy and performance.\n\nQ3: The authors should make clear and mention that they use datasets for LJP in the Chinese language in the context of the People's Republic of China (PRC) jurisdiction (law).\n\nA3: Thank you for your valuable comments. We will update the manuscript to state upfront that our LJP research focuses on Chinese legal documents under the jurisdiction of the People's Republic of China. Phrases like \"LJP aims to predict...\" will be rephrased to indicate we are referring specifically to the Chinese LJP task, rather than implying it is universally applicable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The authors propose an interesting framework relying on contrastive learning to \"push\" the predictive performance in the Chinese LJP multi-task benchmarks.", "reasons_to_reject": "-  The proposed framework is built around a specific setting (3-task Chinese LJP), and there is no evidence that a similar methodology would be useful in other related LJP tasks from other jurisdictions (Chalkidis et al., 2019; Niklaus et al., 2021; and others).", "questions_to_authors": "- The authors mention that using standard cross-entropy is not an ideal solution, since \"The model should be punished more if it classifies a charge into its corresponding confusing charges.\". This sounds counter-intuitive to me since misclassifications naturally occur in closed-related subjects (close-related labels, e.g., similar types of crimes). I don't understand why such misclassifications should be penalized more compared to misclassifications in favor of very unrelated crimes. Intuitively, misclassifying the crime of \"Picking a Quarrel\" as \"Intentional Injury\" sounds less problematic compared to picking a much more unrelated crime such as \"Drug Trafficking\" for instance. Can you explain the reasoning (intuition) behind the proposed solution?", "paper_topic_and_main_contributions": "The authors propose a new framework for Legal Judgment Prediction (LJP) for the Chinese CAIL multi-task datasets, where a model has to predict the relevant criminal charge, law articles, and penalty term based on criminal case facts. The authors propose (a) the use of contrastive learning to better discriminate charges, and (b) the use of a numerical evidence extractor to enhance the penalty term predictor.\nThe authors experiment with two datasets (CAIL-Small, and CAIL-Big), and compare the use of the framework to several other baselines (HARNN, LADAN, NeurJudge+, CrimeBERT). The absolute performance improvement for using the proposed framework is approx. 1-1.5% across all methods.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- The authors should make clear and mention that they use datasets for LJP in the Chinese language in the context of People's Republic of China (PRC) jursdiction (law). There are phrases like \"LJP aims to predict its charge, applicable law article, and term of penalty.\", which sound like the LJP is a universal task across languages and countries. The Bender Rule: \"Always name the language you're working on\" is not only applicable to English."}}]

Author rebuttal:

Meta-review: The proposed methodology capitalizes on the numerical data embedded within LJP texts, demonstrating a notable improvement in performance.
The authors innovatively apply a momentum contrast-based supervised contrastive learning approach, addressing the issue of large class numbers and the subsequent difficulty in identifying sufficient negative examples within mini-batches.
The introduced framework exhibits versatility, being adaptable to current encoder-based methodologies.

The proposed framework is built around a specific setting (3-task Chinese LJP), and there is no evidence that a similar methodology would be useful in other related LJP tasks from other jurisdictions (Chalkidis et al., 2019; Niklaus et al., 2021; and others).
Decision: Accept-Findings