CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain

Abstract: In this paper, we introduce CAIL2019-SCM, Chinese AI and Law 2019 Similar Case Matching dataset. CAIL2019-SCM contains 8,964 triplets of cases published by the Supreme People's Court of China. CAIL2019-SCM focuses on detecting similar cases, and the participants are required to check which two cases are more similar in the triplets. There are 711 teams who participated in this year's competition, and the best team has reached a score of 71.88. We have also implemented several baselines to help researchers better understand this task. The dataset and more details can be found from this https URL.

Similar Case Matching (SCM) plays a major role in legal system, especially in common law legal system. The most similar cases in the past determine the judgment results of cases in common law systems. As a result, legal professionals often spend much time finding and judging similar cases to prove fairness in judgment. As automatically finding similar cases can benefit to the legal system, we select SCM as one of the tasks of CAIL2019.
Chinese AI and Law Challenge (CAIL) is a competition of applying artificial intelligence technology to legal tasks. The goal of the competition is to use AI to help the legal system. CAIL was first held in 2018, and the main task of CAIL2018 BIBREF0, BIBREF1 is predicting the judgment results from the fact description. The judgment results include the accusation, applicable articles, and the term of penalty. CAIL2019 contains three different tasks, including Legal Question-Answering, Legal Case Element Prediction, and Similar Case Matching. Furthermore, we will focus on SCM in this paper.
More specifically, CAIL2019-SCM contains 8,964 triplets of legal documents. Every legal documents is collected from China Judgments Online. In order to ensure the similarity of the cases in one triplet, all selected documents are related to Private Lending. Every document in the triplet contains the fact description. CAIL2019-SCM requires researchers to decide which two cases are more similar in a triplet. By detecting similar cases in triplets, we can apply this algorithm for ranking all documents to find the most similar document in the database. There are 247 teams who have participated CAIL2019-SCM, and the best team has reached a score of $71.88$, which is about 20 points higher than the baseline. The results show that the existing methods have made great progress on this task, but there is still much room for improvement.
In other words, CAIL2019-SCM can benefit the research of legal case matching. Furthermore, there are several main challenges of CAIL2019-SCM: (1) The difference between documents may be small, and then it is hard to decide which two documents are more similar. Moreover, the similarity is defined by legal workers. We must utilize legal knowledge into this task rather than calculate similarity on the lexical level. (2) The length of the documents is quite long. Most documents contain more than 512 characters, and then it is hard for existing methods to capture document level information.
In the following parts, we will give more details about CAIL2019-SCM, including related works about SCM, the task definition, the construction of the dataset, and several experiments on the dataset.
We first define the task of CAIL2019-SCM here. The input of CAIL2019-SCM is a triplet $(A,B,C)$, where $A,B,C$ are fact descriptions of three cases. Here we define a function $sim$ which is used for measuring the similarity between two cases. Then the task of CAIL2019-SCM is to predict whether $sim(A,B)>sim(A,C)$ or $sim(A,C)>sim(A,B)$.
To ensure the quality of the dataset, we have several steps of constructing the dataset. First, we select many documents within the range of Private Lending. However, although all cases are related to Private Lending, they are still various so that many cases are not similar at all. If the cases in the triplets are not similar, it does not make sense to compare their similarities. To produce qualified triplets, we first annotated some crucial elements in Private Lending for each document. The elements include:
The properties of lender and borrower, whether they are a natural person, a legal person, or some other organization.
The type of guarantee, including no guarantee, guarantee, mortgage, pledge, and others.
The usage of the loan, including personal life, family life, enterprise production and operation, crime, and others.
The lending intention, including regular lending, transfer loan, and others.
Conventional interest rate method, including no interest, simple interest, compound interest, unclear agreement, and others.
Interest during the agreed period, including $[0\%,24\%]$, $(24\%,36\%]$, $(36\%,\infty )$, and others.
Borrowing delivery form, including no lending, cash, bank transfer, online electronic remittance, bill, online loan platform, authorization to control a specific fund account, unknown or fuzzy, and others.
Repayment form, including unpaid, partial repayment, cash, bank transfer, online electronic remittance, bill, unknown or fuzzy, and others.
Loan agreement, including loan contract, or borrowing, “WeChat, SMS, phone or other chat records”, receipt, irrigation, repayment commitment, guarantee, unknown or fuzzy and others.
After annotating these elements, we can assume that cases with similar elements are quite similar. So when we construct the triplets, we calculate the tf-idf similarity and elemental similarity between cases and select those similar cases to construct our dataset. We have constructed 8,964 triples in total by these methods, and the statistics can be found from Table TABREF13. Then, legal professionals will annotate every triplet to see whether $sim(A,B)>sim(A,C)$ or $sim(A,B)<sim(A,C)$. Furthermore, to ensure the quality of annotation, every document and triplet is annotated by at least three legal professionals to reach an agreement.
In this paper, we propose a new dataset, CAIL2019-SCM, which focuses on the task of similar case matching in the legal domain. Compared with existing datasets, CAIL2019-SCM can benefit the case matching in the legal domain to help the legal partitioners work better. Experimental results also show that there is still plenty of room for improvement.

Figure and table captions:
Table 1: The number of triplets in different stages of CAIL2019-SCM.
Table 2: Results of baselines and scores of top 3 participants on valid and test datasets.

Questions about this paper:
What are the baselines?

Answers:
CNN, LSTM, BERT