SlowGuess/ACL_2020_OCR huggingface dataset:
Nuo Xu $^{1}$ , Pinghui Wang $^{2,1*}$ , Long Chen $^{1}$ , Li Pan $^{3}$ , Xiaoyan Wang $^{4}$ , Junzhou Zhao $^{1*}$
$^{1}$ MOE NEKEY Lab, Xi'an Jiaotong University, China
$^{2}$ Shenzhen Research School, Xi'an Jiaotong University, China
$^{3}$ School of Electronic, Information and Electrical Engineering, Shanghai Jiao Tong University
<sup>4</sup>Information Technology Service Center, The Supreme People's Court, China
nxu@sei.xjtu.edu.cn, phwang@mail.xjtu.edu.cn,
chenlongche@stu.edu.cn, panli@sjtu.edu.cn,
wangxiaoyan@court.gov.cn, junzhouzhao@gmail.com
Legal Judgment Prediction (LJP) is the task of automatically predicting a law case's judgment results given a text describing its facts, which has excellent prospects in judicial assistance systems and convenient services for the public. In practice, confusing charges are frequent, because law cases applicable to similar law articles are easily misjudged. For addressing this issue, the existing method relies heavily on domain experts, which hinders its application in different law systems. In this paper, we present an end-to-end model, LADAN, to solve the task of LJP. To distinguish confusing charges, we propose a novel graph neural network to automatically learn subtle differences between confusing law articles and design a novel attention mechanism that fully exploits the learned differences to extract compelling discriminative features from fact descriptions attentively. Experiments conducted on real-world datasets demonstrate the superiority of our LADAN.
Exploiting artificial intelligence techniques to assist legal judgment has become popular in recent years. Legal judgment prediction (LJP) aims to predict a case's judgment results, such as applicable law articles, charges, and terms of penalty, based on its fact description, as illustrated in Figure 1. LJP can assist judiciary workers in processing cases and offer legal consultancy services to the public. In the literature, LJP is usually formulated as a text classification problem, and several rule-based methods (Liu et al., 2004; Lin et al., 2012) and neural-based methods (Hu et al., 2018; Luo et al., 2017; Zhong et al., 2018) have been proposed.
The main drawback of existing methods is that they cannot solve the confusing charges issue.
Figure 1: An illustration of the LJP. Generally, a judge needs to conduct professional analysis and reasoning on the fact description of the case, and then choose reasonable law articles, charges and the term of penalty to convict the offender.
That is, due to the high similarity of several law articles, their corresponding law cases can be easily misjudged. For example, in Figure 2, both Article 385 and Article 163 describe offenses of accepting bribes, and their subtle difference is whether the guilty parties are state staffs or not. The key to solving the confusing charges issue is how to capture essential but rare features for distinguishing confusing law articles. Hu et al. (2018) defined ten discriminative attributes to distinguish confusing charges. However, their method relies too much on experts to hinder its applications in a large number of laws. In practice, we desire a method that can automatically extract textual features from law articles to assist JLP. The most relevant existing work to this requirement is (Luo et al., 2017), which used an attention mechanism to extract features from fact descriptions with respect to a specific law article. As shown in Figure 3a, for each law article, an attention vector is computed, which is used to extract features from the fact description of a law case to predict whether the law article is applicable to the case. Nevertheless, the weakness
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3086-3095
Article 385: The Crime of acceptance of bribes
Any state staffs who, taking advantage of his position, demands money or property from another person, or illegally accepts another person's money or property in return for securing benefits for the person shall be guilty of acceptance of bribes.
Article 163: Bribery crime of non-state staffs
The employees of companies, enterprises or other units who, taking advantage of his position, demands money or property from another person, or illegally accepts another person's money or property in return for securing benefits for the person shall be guilty of bribery crime of non-state staffs.
Article 389: Crime of offering bribes
Whoever, in order to seek illegitimate benefits, gives any state staffs with money and property, shall be the crime of bribery
Article 164: The crime of offering bribes to non-state staff
Whoever, in order to seek illegitimate benefits, gives employees of companies, enterprises or other units with money or property, shall be guilty of bribing non-state staffs.
is that they learn each law article's attention vector independently, and this may result in that similar attention vectors are learned for semantically close law articles; hence, it is ineffective in distinguishing confusing charges.
To solve the confusing charges issue, we propose an end-to-end framework, i.e., Law Article Distillation based Attention Network (LADAN). LADAN uses the difference among similar law articles to attentively extract features from law cases' fact descriptions, which is more effective in distinguishing confusing law articles, and improve the performance of LJP. To obtain the difference among similar law articles, a straightforward way is to remove duplicated texts between two law articles and only use the leftover texts for the attention mechanism. However, we find that this method may generate the same leftover texts for different law article, and generate misleading information to LJP. As shown in Fig. 2, if we remove the duplicated phrases and sentences between Article 163 and Article 385 (i.e., the red text in Fig. 2), and between Article 164 and Article 389 (i.e., the pink text in Fig. 2), respectively, then Article 385 and Article 389 will be almost same to each other (i.e., the blue text in Fig. 2).
We design LADAN based on the following observation: it is usually easy to distinguish dissimilar law articles as sufficient distinctions exist, but challenging to discriminate similar law articles due to the few useful features. We first group law articles into different communities, and law articles in the same community are highly similar to each other. Then we propose a graph-based representation learning method to automatically explore the difference among law articles and com
Figure 2: Examples of confusing charges.
Figure 3: a. The fact-law attention model in (Luo et al., 2017). b. Our framework. Variables  $\alpha$  and  $\beta$  represent the encoded vectors learned for attentively extracting features from fact descriptions.
pute an attention vector for each community. For an input law case, we learn both macro- and micro-level features. Macro-level features are used for predicting which community includes the applicable law articles. Micro-level features are attentively extracted by the attention vector of the selected community for distinguishing confusing law articles within the same community. Our main contributions are summarized as follows:
(1) We develop an end-to-end framework, i.e., LADAN, to solve the LJP task. It addresses the confusing charges issue by mining similarities between fact descriptions and law articles as well as the distinctions between confusing law articles.
(2) We propose a novel graph distillation operator (GDO) to extract discriminative features for effectively distinguishing confusing law articles.
(3) We conduct extensive experiments on real-world datasets. The results show that our model outperforms all state-of-the-art methods.
Our work solves the problem of the confusing charge in the LJP task by referring to the calculation principle of graph neural network (GNN). Therefore, in this section, we will introduce related works from these two aspects.
2.1 Legal Judgment Prediction
Existing approaches for legal judgment prediction (LJP) are mainly divided into three categories. In early times, works usually focus on analyzing existing legal cases in specific scenarios with mathematical and statistical algorithms (Kort, 1957; Nagel, 1963; Keown, 1980; Lauderdale and Clark, 2012). However, these methods are limited to small datasets with few labels. Later, a number of
machine learning-based methods (Lin et al., 2012; Liu et al., 2004; Sulea et al., 2017) were developed to solve the problem of LJP, which almost combine some manually designed features with a linear classifier to improve the performance of case classification. The shortcoming is that these methods rely heavily on manual features, which suffer from the generalization problem.
In recent years, researchers tend to exploit neural networks to solve LJP tasks. Luo et al. (2017) propose a hierarchical attentional network to capture the relation between fact description and relevant law articles to improve the charge prediction. Zhong et al. (2018) model the explicit dependencies among subtasks with scalable directed acyclic graph forms and propose a topological multi-task learning framework for effectively solving these subtasks together. Yang et al. (2019) further refine this framework by adding backward dependencies between the prediction results of subtasks. To the best of our knowledge, Hu et al. (2018) are the first to study the problem of discriminating confusing charges for automatically predicting applicable charges. They manually define 10 discriminative attributes and propose to enhance the representation of the case fact description by learning these attributes. This method relies too much on experts and cannot be easily extended to different law systems. To solve this issue, we propose a novel attention framework that automatically extracts differences between similar law articles to enhance the representation of fact description.
2.2 Graph Neural Network
Due to its excellent performance in graph structure data, GNN has attracted significant attention (Kipf and Welling, 2017; Hamilton et al., 2017; Bonner et al., 2019). In general, existing GNNs focus on proposing different aggregation schemes to fuse features from the neighborhood of each node in the graph for extracting richer and more comprehensive information: Kipf et al. (2017) propose graph convolution networks which use mean pooling to pool neighborhood information; GraphSAGE (Hamilton et al., 2017) concatenates the node's features and applies mean/max/LSTM operators to pool neighborhood information for inductively learning node embeddings; MR-GNN (Xu et al., 2019) aggregates the multi-resolution features of each node to exploit n-node information, subgraph information, and glob
al information together; Besides, Message Passing Neural Networks (Gilmer et al., 2017) further consider edge information when doing the aggregation. However, the aggregation schemes lead to the over-smoothing issue of graph neural networks (Li et al., 2018), i.e., the aggregated node representations would become indistinguishable, which is entirely contrary to our goal of extracting distinguishable information. So in this paper, we propose our distillation operation, based on a distillation strategy instead of aggregation schemes, to extract the distinguishable features between similar law articles.
In this section, we introduce some notations and terminologies, and then formulate the LJP task.
Law Cases. Each law case consists of a fact description and several judgment results (cf. Figure 1). The fact description is represented as a text document, denoted by  $f$ . The judgment results may include applicable law articles, charges, terms of penalty, etc. Assume there are  $t$  kinds of judgment results, and the  $i$ -th judgment result is represented as a categorical variable  $y_{i}$  which takes value from set  $Y_{i}$ . Then, a law case can be represented by a tuple  $(f, y_{1}, \ldots, y_{t})$ .
Law Articles. Law cases are often analyzed and adjudicated according to a legislature's statutory law (also known as, written law). Formally, we denote the statutory law as a set of law articles  $\mathcal{L} \triangleq \{L_1, \ldots, L_m\}$  where  $m$  is the number of law articles. Similar to the fact description of cases, we also represent each law article  $L_i$  as a document.
Legal Judgment Prediction. In this paper, we consider three kinds of judgment results: applicable law articles, charges, and terms of penalty. Given a training dataset  $D \triangleq \{(f, y_1, y_2, y_3)_z\}_{z=1}^q$  of size  $q$ , we aim to train a model  $\mathrm{F}(\cdot)$  that can predict the judgment results for any test law case with a fact description  $f_{test}$ , i.e.,  $\mathrm{F}(f_{test}, \mathcal{L}) = (\hat{y}_1, \hat{y}_2, \hat{y}_3)$ , where  $\hat{y}_i \in Y_i$ ,  $i = 1, 2, 3$ . Following (Zhong et al., 2018; Yang et al., 2019), we assume each case has only one applicable law article.
Figure 4: a. Overview of our framework LADAN: it takes the fact descriptions of cases and the text definitions of law articles as inputs, then extracts the basic representation  $\mathbf{v}_f^{\mathrm{b}}$  and distinguishing representation  $\mathbf{v}_f^{\mathrm{d}}$  of the fact descriptions through the basic encoder and the re-encoder, and finally combines this two representations for the downstream prediction tasks; b. Law Distillation Module: this module communicates law articles and distills the distinguishable features of each community for attention calculation of the re-encoder.
In our framework LADAN (cf. Fig. 4a), the fact description of a case is represented by two parts: a basic representation, denoted by  $\mathbf{v}_f^{\mathrm{b}}$ , and a distinguishable representation, denoted by  $\mathbf{v}_f^{\mathrm{d}}$ . The basic representation  $\mathbf{v}_f^{\mathrm{b}}$  contains basic semantic information for matching a group of law articles that may apply to the case. In contrast, the distinguishable representation  $\mathbf{v}_f^{\mathrm{d}}$  captures features that can effectively distinguish confusing law articles. The concatenation of  $\mathbf{v}_f^{\mathrm{b}}$  and  $\mathbf{v}_f^{\mathrm{d}}$  is fed into subsequent classifiers to predict the labels of the JLP task.
As we mentioned, it is easy to distinguish dissimilar law articles as sufficient distinctions exist, and the difficulty in solving confusing charges lies in extracting distinguishable features of similar law articles. To obtain the basic representation  $\mathbf{v}_f^{\mathrm{b}}$ , therefore, we use one of the popular document encoding methods (e.g., CNN encoder (Kim, 2014) and Bi-RNN encoder (Yang et al., 2016)). To learn the distinguishable representation  $\mathbf{v}_f^{\mathrm{d}}$ , we use a law distillation module first to divide law articles to several communities to ensure that the law articles in each community are highly similar, and then extract each community  $i$ 's distinction vector
(or, distinguishable features)  $\beta_{i}$  from the basic representation of law articles in community  $i$ . Given the case's fact description, from all communities' distinction vectors, we select the most relevant one (i.e.,  $\beta_{\hat{c}}$  in Fig. 4(a)) for attentively extracting the distinguishable features  $\mathbf{v}_f^{\mathrm{d}}$  in the fact re-encode module. In the follows, we elaborate law distillation module (Sec. 4.2) and fact re-encode module (Sec. 4.3) respectively.
4.2 Distilling Law Articles
A case might be misjudged due to the high similarity of some law articles. To alleviate this problem, we design a law distillation module (cf. Fig. 4 b) to extract distinguishable and representative information from all law articles. Specifically, it first uses a graph construction layer (GCL) to divide law articles into different communities. For each law article community, a graph distillation layer is applied to learn its discriminative representation, hereinafter, called distinction vector.
4.2.1 Graph Construction Layer
To find probably confusing law articles, we first construct a fully-connected graph  $G^{*}$  for all law articles  $\mathcal{L}$ , where the weight on the edge between a pair of law article  $L_{i}, L_{j} \in \mathcal{L}$  is defined as
the cosine similarity between their TF-IDF (Term Frequency-Inverse Document Frequency) representations  $\mathbf{tf\_idf}_i$  and  $\mathbf{tf\_idf}_j$ . Since confusing law articles are usually semantically similar and there exists sufficient information to distinguish dissimilar law articles, we remove the edges with weights less than a predefined threshold  $\tau$  from graph  $G^{*}$ . By setting an appropriate  $\tau$ , we obtain a new graph  $G = \{g_i\}_{i=1}^M$  composed of several disconnected subgraphs  $g_1, \ldots, g_M$  (or, communities), where each  $g_i, i = 1, \ldots, M$  contains a specific community of probably confusing articles. Our later experimental results demonstrate that this easy-to-implement method effectively improves the performance of LADAN.
4.2.2 Graph Distillation Layer
To extract the distinguishable information from each community  $g_{i}$ , a straightforward way is to delete duplicate words and sentences presented in law articles within the community (as described in Sec. 1). In addition to introducing significant errors, this simple method cannot be plugged into end-to-end neural architectures due to its non-differentiability. To overcome the above issues, inspired by the popular graph convolution operator (GCO) (Kipf and Welling, 2017; Hamilton et al., 2017; Velicković et al., 2017), we propose a graph distillation operator (GDO) to effectively extract distinguishable features. Different from GCO, which computes the message propagation between neighbors and aggregate these messages to enrich representations of nodes in the graph, the basic idea behind our GDO is to learn effective features with distinction by removing similar features between nodes.
Specifically, for an arbitrary law article  $L_{i}$ , GDO uses a trainable weight matrix  $\Psi$  to capture similar information between it and its neighbors in graph  $G$ , and a matrix  $\Phi$  to extract effective semantic features of  $L_{i}$ . At each layer  $l \geq 0$ , the aggregation of similar information between  $L_{i}$  and its neighbors is removed from its representation, that is,
$$
\mathbf {v} _ {L _ {i}} ^ {(l + 1)} = \boldsymbol {\Phi} ^ {(l)} \mathbf {v} _ {L _ {i}} ^ {(l)} - \sum_ {L _ {j} \in N _ {i}} \frac {\boldsymbol {\Psi} ^ {(l)} [ \mathbf {v} _ {L _ {i}} ^ {(l)} , \mathbf {v} _ {L _ {j}} ^ {(l)} ]}{| N _ {i} |} + \mathbf {b} ^ {(l)}
$$
where  $\mathbf{v}_{L_i}^{(l)}\in \mathbb{R}^{d_l}$  refers to the representation of law  $L_{i}$  in the  $l^{\mathrm{th}}$  graph distillation layer,  $N_{i}$  refers to the neighbor set of  $L_{i}$  in graph  $G$ ,  $\mathbf{b}^{(l)}$  is the
bias, and  $\Phi^{(l)}\in \mathbb{R}^{d_{l + 1}\times d_l}$  and  $\Psi^{(l)}\in \mathbb{R}^{d_{l + 1}\times 2d_l}$  are the trainable self weighted matrix and the neighbor similarity extracting matrix respectively. Note that  $d_{l}$  is the dimension of the feature vector in the  $l^{\mathrm{th}}$  graph distillation layer. We set  $d_0 = d_s$  where  $d_{s}$  is the dimension of basic representations  $\mathbf{v}_f^{\mathrm{b}}$  and  $\mathbf{v}_{L_i}$ . Similar to GCO, our GDO also supports multi-layer stacking.
Using GDO with  $H$  layers, we output law article representation of the last layer, i.e.,  $\mathbf{v}_{L_i}^{(H)}\in \mathbb{R}^{d_H}$ , which contains rich distinguishable features that can distinguish law article  $L_{i}$  from the articles within the same community. To further improve law articles' distinguishable features, for each subgraph  $g_{i},i = 1,2,\ldots ,M$  in graph  $G$ , we compute its distinction vector  $\beta_{i}$  by using pooling operators to aggregate the distinguishable features of articles in  $g_{i}$ . Formally,  $\beta_{i}$  is computed as:
$$
\beta_ {i} = \left[ \mathrm {M a P} \left(\left\{\mathbf {v} _ {L _ {i}} ^ {(H)} \right\} _ {L _ {j} \in g _ {i}}\right), \mathrm {M i P} \left(\left\{\mathbf {v} _ {L _ {i}} ^ {(H)} \right\} _ {L _ {j} \in g _ {i}}\right) \right]
$$
where  $\mathrm{MaP}(\cdot)$  and  $\mathrm{MiP}(\cdot)$  are the element-wise max pooling and element-wise min pooling operators respectively.
4.3 Re-encoding Fact with Distinguishable Attention
To capture a law case's distinguishable features from its fact description  $f$ , we firstly define the following linear function, which is used to predict its most related community  $g_{\hat{c}}$  in graph  $G$ :
$$
\hat {\mathbf {X}} = \operatorname {s o f t m a x} \left(\mathbf {W} _ {g} \mathbf {v} _ {f} ^ {\mathrm {b}} + \mathbf {b} _ {g}\right) \tag {1}
$$
where  $\mathbf{v}_f^{\mathrm{b}}$  is the basic representation of fact description  $f$ ,  $\mathbf{W}_g \in \mathbb{R}^{M \times d_s}$  and  $\mathbf{b}_g \in \mathbb{R}^M$  are the trainable weight matrix and bias respectively. Each element  $\hat{X}_i \in \hat{\mathbf{X}}$ ,  $i = 1, \dots, M$  reflects the closeness between fact description  $f$  and law articles community  $g_i$ . The most relevant community  $g_{\hat{c}}$  is computed as
$$
\hat{c} = \arg \max_{i = 1,\ldots ,M}\hat{X}_{i}.
$$
Then, we use the corresponding community's distinction vector  $\beta_{\hat{c}}$  to attentively extract distinguishable features from fact description  $f$ .
Inspired by (Yang et al., 2016), we attentively extract distinguishable features based on word-level and sentence-level Bi-directional Gated Recurrent Units (Bi-GRUs). Specifically, for each input sentence  $S_{i} = [w_{i,1},\dots ,w_{i,n_{i}}]$  in fact description  $f$ , word-level Bi-GRUs will output a hidden
state sequence, that is,
$$
\mathbf {h} _ {i, j} = \overleftarrow {\mathrm {G R U}} (\mathbf {w} _ {i, j}), \overleftarrow {\mathrm {G R U}} (\mathbf {w} _ {i, j}) ], j = 1,..., n _ {i},
$$
where  $\mathbf{w}_{i,j}$  represents the word embedding of word  $w_{i,j}$  and  $\mathbf{h}_{i,j} \in \mathbb{R}^{d_w}$ . Based on this hidden state sequence and the distinction vector  $\beta_{\hat{c}}$ , we calculate an attentive vector  $[\alpha_{i,1}, \ldots, \alpha_{i,n_i}]$ , where each  $\alpha_{i,j}$  evaluates the discrimination ability of word  $w_{i,j} \in S_i$ .  $\alpha_{i,j}$  is formally computed as:
$$
\alpha_ {i, j} = \frac {\exp (\tanh (\mathbf {W} _ {w} \mathbf {h} _ {i , j}) ^ {\top} (\mathbf {W} _ {g w} \boldsymbol {\beta} _ {\hat {c}}))}{\sum_ {j} \exp (\tanh (\mathbf {W} _ {w} \mathbf {h} _ {i , j}) ^ {\top} (\mathbf {W} _ {g w} \boldsymbol {\beta} _ {\hat {c}}))},
$$
where  $\mathbf{W}_w$  and  $\mathbf{W}_{gw}$  are trainable weight matrices. Then, we get a representation of sentence  $S_{i}$  as:
$$
\mathbf {v} _ {s _ {i}} = \sum_ {j = 1} ^ {n _ {i}} \alpha_ {i, j} \mathbf {h} _ {i, j},
$$
where  $n_i$  denotes the word number in sentence  $S_i$ .
By the above word-level Bi-GRUs, we get a sentence representations sequence  $\left[\mathbf{v}_{s_1},\ldots ,\mathbf{v}_{s_{n_f}}\right]$ , where  $n_f$  refers to the number of sentences in the fact description  $f$ . Based on this sequence, similarly, we build sentence-level Bi-GRUs and calculate a sentence-level attentive vector  $[\alpha_{1},\dots,\alpha_{n_{f}}]$  that reflects the discrimination ability of each sentence, and then get the fact's distinguishable representation  $\mathbf{v}_f^{\mathrm{d}}\in \mathbb{R}^{d_s}$ . Our sentence-level Bi-GRUs are formulated as:
$$
\mathbf {h} _ {i} = [ \overrightarrow {\mathrm {G R U}} (\mathbf {v} _ {s _ {i}}), \overleftarrow {\mathrm {G R U}} (\mathbf {v} _ {s _ {i}}) ], i = 1, 2, \dots , n _ {f},
$$
$$
\alpha_ {i} = \frac {\exp (\tanh (\mathbf {W _ {s} h _ {i}}) ^ {\top} (\mathbf {W _ {g s} \beta_ {\hat {c}}}))}{\sum_ {i} \exp (\tanh (\mathbf {W _ {s} h _ {i}}) ^ {\top} (\mathbf {W _ {g s} \beta_ {\hat {c}}}))},
$$
$$
\mathbf {v} _ {f} ^ {\mathrm {d}} = \sum_ {i} \alpha_ {i} \mathbf {h} _ {i}.
$$
4.4 Prediction and Training