Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a Korean language legal NLP dataset, LBOX OPEN, and a Korean legal language model trained on LBOX OPEN, LCUBE, a decoder only model based on a GPT-2 architecture. The dataset includes a legal precedent corpus, two multi-class classification tasks, two legal judgment prediction tasks (with quantized ground-truth predictions) covering criminal and civil cases, and one legal summarization task.

The authors show that pretraining on a domain specific corpus result in performance gains across several of the tasks and comparable performance to a model with more capacity. They find that the domain specific pretraining is less helpful for the legal summarization task. These results are consistent with existing results on English language legal datasets and pretraining corpora. The originality comes primarily from the creation and evaluation of legal datasets and pretraining corpora in Korean.
strengths: There are few pretraining corpora / datasets and legal-domain specific language models for Korean law. This paper introduces a large-scale Korean language legal corpus, different downstream NLP tasks relevant to the legal domain, such as legal judgment prediction, and a pretrained legal-domain language model, which I believe will be valuable in providing more multi-lingual NLP resources for the legal domain. Additionally, the dataset includes tasks over a diverse range of downstream tasks that are of interest to the legal NLP community, including more difficult tasks such as summarization that are often expensive to collect ground-truth labels for.

The authors put a significant amount of engineering effort and human validation into extracting structure data from raw data, since most Korean legal documents are available in either non-digitized document image or PDF formats.

Details were provided that would make it possible to reproduce pretraining for LCUBE-base and the fine-tuning procedure used for downstream tasks. I thought the experiments removing the facts from input were useful in showing the importance of the facts in the model reasoning about judgment predictions.

The data and language model are accessible through HuggingFace. See additional comments in Documentation section on potential improvements to accessibility and accountability.
weaknesses: I would have liked to see a little more justification about some of the quantization choices to form classification tasks out of the two legal judgment prediction tasks. It seemed like some different choices were made about maintaining class balance in quantization for the criminal vs. civil legal judgment prediction tasks and it was not entirely clear to me from the experiments whether the results would be affected by the somewhat subjective quantization decisions.

It might be helpful to discuss potentially reasons that the summarization task does not see improvements from legal domain-specific pretraining.

There is limited discussion of the social and ethical impacts of the dataset. I think two specific points that should be included are (1) the dataset’s license and whether any of the incorporated data from other sources is under other licenses, (2) whether there is potentially personal identifiable information in the legal case corpus and the norms / privacy protections governing Korean legal data released by the government and legal system.
correctness: In terms of experimentation, I thought the experiment choices were reasonable and illustrative, with the LCUBE-base domain-specific pretrained language model compared against majority class and a general domain pretrained Korean language model with a comparable GPT-2 architecture.

It seemed that while it is true that compared to the other tasks, the gain in performance from domain-specific pretraining was smaller for the legal judgment prediction tasks, it was unclear to me whether this can be completely attributed to legal judgment tasks being inherently more challenging, as described in the second paragraph of the Results section. It appears that the gain from domain-specific pretraining is similar for LJP-CIVIL and CASE NAME, but the gain for LJP-CRIMINAL is much smaller. Could this be due to the number of classes being greater / the quantization being more granular or the classes being more balanced for the criminal LJP tasks? The share of LJP-CIVIL examples with label 2 is so small that the task simplifies somewhat to something close to a binary classification task, while the LJP-CRIMINAL tasks were quantized with consideration for both class balance and context specific legal aspects. I would have liked to see more justification for this or perhaps some experimentation that shows the somewhat subjective / human determined quantization doesn’t affect results significantly. What happens when the LJP tasks are modeled as regression tasks? Even if some of the predictive factors are not available (e.g., age), so performance might be expected to be worse, do the same relative differences in gains still hold?
clarity: Overall, the paper was clear and easy to follow throughout. 

I think the gains achieved from domain-specific pretraining (row 1 vs. row 2 in the table on page 7) can be pulled up and summarized more clearly earlier in the paper, in paragraph 3 on page 2.

There were a few small typos:
In Abstract, “the” missing right before “legal domain” in the third line.
Pg 2, “in last”->”in the last” in paragraph 2, line 5
Pg 5, Table 3, LJP-CIVIL subtask, “large-scalethe degrees”, replace with “the degree”?
Pg 5, “fine-tnue”->”fine-tune” in 4.1 Model training, line 2
Pg 6, missing “the” before “following” in paragraph 1, line 1
Pg 6, “archtiecture”->”architecture” Results, paragraph 2, line 2
Pg 7, “gits”->”gists” in paragraph 2, line 2
relation to prior work: I think the methodology is similar to those of other recent English language legal NLP papers, but the paper clearly outlines that it differs in that it is providing Korean language datasets and pretrained language models for the legal domain, which has not been done at this scale in other work.
additional feedback: My main points of feedback are:
- More thorough justification or experimentation on the effects of label quantization counterfactuals for the legal judgment prediction tasks, which seemed somewhat subjective
- More detailed discussion of the licensing / distribution / use restrictions and if / where there may be personally identifiable information, which can in especially prevalent in some types of legal documents (and if so, elaboration on mitigation strategies or rationale for releasing dataset as is)
review_title: Official Review of Paper92 by Reviewer LvmP

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The authors present a new benchmark for three different types of legal tasks (classification, judgement prediction, and summarization) in Korean. Specifically, they introduce six datasets: (1) a legal precedent corpus consisting of 150k precedents; (2) a multi-class legal text classification dataset consisting of 10,000 facts-category pairs with 100 different case categories; (3) another multi-class legal text classification dataset consisting of 2,760 facts-statutes pairs; (4) a legal judgement prediction dataset consisting of 11,000 cases from 7 categories; (5) another legal judgement prediction dataset consisting of 4,678 pairs of facts and degree of money claim acceptance; and (6) a summarization dataset consisting of 20,000 precedent-summary pairs. They also released a GPT2-like model for the Korean legal language and fine-tuned it on the different datasets.
strengths: First such large-scale domain-specific benchmark in Korean with good documentation. Existing legal tasks are largely English-oriented. Having legal NLP tasks in a diverse set of languages is important.
weaknesses: * Apart from the specificity of the language, the motivation for this new benchmark is not explicit. The authors propose several legal tasks based on the data they were able to collect but do not emphasize the importance of these tasks in the real world and how they could benefit society (apart from the summarization task, perhaps).

* As the authors state in Section 3.1, structuring raw data that contains a mix of plain text, tables, and images is a non-trivial task and represents an important part of the work and value provided in the paper. However, little detail is given regarding the different components of the data engineering pipeline (no information about the specific checkpoints or model parameters used for the layout classifier and parser, the OCR framework, and the model used to correct OCR errors). Additionally, code for extracting quality data from the raw documents does not seem publicly released (unlike code for experiments), which is a pity as it would provide excellent value for the research community.

* Authors claim that LCube is a legal language model. However, it turns out that the legal corpora only represent 5.5% of the whole pre-training corpus, whereas news and books represent 90% of the pre-training data. I wonder why the authors did not start from a pre-trained KoGPT-2 checkpoint and further pre-trained on their legal corpus only, which has been shown to lead to similar or even better results than pre-training from scratch for a portion of the computing resources — see ClinicalBERT (Huang et al., 2019), BioBERT (Lee et al., 2020), SciBERT (Beltagy et al., 2019).

Other nits:

* In the Abstract and Introduction, the authors mention some approximative dataset sizes that exceed the actual sizes of the datasets (e.g., 3k pairs for Statute while 8% smaller in reality, 11k pairs for LJP-Criminal while 4.5% smaller in reality, 5k pairs for LJP-Civil while 6.4% smaller in reality). Authors should prefix these approximations with "around"/"~" in order not to mislead the reader about the actual sizes of the provided datasets.

* For LJP-Criminal, the authors state that unavailable data (such as the age of defendants) makes the regression task too difficult and therefore formulate it as a classification task by discretizing the outputs. However, there does not seem to have been a preliminary investigation that supports these claims. Also, the authors explain that the boundaries for quantization are chosen in part based on Korean legal aspects and give the example of 1M won as the lower bound amount where public officials can lose their position if found guilty. This would be interesting to not only have one example of such legal aspects but a clear description of all of them for each bound they correspond to. Additionally, authors should show the data distribution related to those boundaries (to make sure that the models do not simply learn to predict the over-represented classes). Same remark for LJP-Civil.

* The Results section feels like it was written relatively quickly: it lacks detailed analysis and is poorly organized. The authors performed good control experiments that deserve their own subsection with a deeper analysis of the results. Similarly, a more detailed comparison between KoGPT-2 and LCube is missing.
correctness: The dataset construction seems correct but is a bit under-detailed, as stated above.
clarity: The paper is generally well written. There are a few typos in the paper though (e.g., "archtiecture" and "gits" in Section 5).
relation to prior work: * There seems to be no discussion of prior work in the paper, except for a brief description of the broader field of legalNLP in the Introduction section. The paper would benefit from an extensive discussion on other legal-domain benchmarks — such as LexGLUE (Chalkidis et al., 2021), FairLex (Chalkidis et al., 2022), JEC-QA (Zhong et al. 2019), CJRC (Duan et al., 2019), Swiss-Judgment-Prediction (Niklaus et al., 2021) — in order to help the reader better understand how this dataset may relate to the other benchmarks.
* Additionally, the authors could cite two valuable resources related to legal information retrieval in French and Japanese, respectively: (i) "A Statutory Article Retrieval Dataset in French" (https://aclanthology.org/2022.acl-long.468/); (ii) "COLIEE 2020: Methods for Legal Document Retrieval and Entailment" (https://link.springer.com/chapter/10.1007/978-3-030-79942-7_13).
additional feedback: * The Korean legal system explanation (Section 2.1) lacks clarity about the difference between civil and common law and the concept of stare decisis. The reader (who is expected to have very little legal knowledge) might benefit from a more detailed explanation of those topics. Same remark for the "no penalty without a law" principle, mentioned at page 4.
* "In the real world, automatic categorization of legal documents or questions are often important for various downstream tasks" (p.4). Authors should give a few concrete examples of such important downstream tasks.
* References to "Appendix B.x" actually correspond to "Appendix A.1.x" in the paper.
* Regarding the statute prediction task, it is mentioned (p.4) that the dataset is built from the same precedents used in CaseName. However, the Statute dataset only contains samples from 46 frequent case categories with 60 examples each, unlike the 100 most frequent case categories with 100 examples each from CaseName. The authors do not explain the choice of reducing both the number of case categories and examples per category for that second text classification task. Also, I assume that the 169 classes stated in Table 3 correspond to the number of different statutes found in those 46*60=2760 samples. If so, it would not be wrong to mention it for clarity.
* For LJP-Civil, the authors explain that they performed the extraction of the amounts of money by "preparing their custom language model trained on a few hundreds of examples of ruling/claim-money pairs. The sentence is very unclear. What do you mean by "preparing", did you fine-tuned LCube on a regression task using these samples?
* For Summarization, the authors do not mention how the summaries were generated. Did the legal experts from the Korean Law Information Center write them? Also, it is a pity that the authors did limit their collection to precedents whose ruling and reasoning sections were less than 1024 tokens, their choice being based on the maximum input length of a specific model at the expense of a more complete dataset containing longer sequences (that could easily be handled with simple workarounds, such as chunking, or with other models allowing longer inputs).
* In Section 4.2, a more detailed explanation regarding the formulation of a text classification task using a generative model is expected. Moreover, the authors refer to the T5 paper while the task formulation is necessarily different between an encoder-decoder and their decoder-only model.
* In Section 4.3, the authors forget to mention which metrics are used for summarization (namely, Rouge).
* The five first sentences of the last paragraph in Section 5 are a repetition of what has already been explained in Section 3.2.
review_title: A useful resource, good documentation, with some missing points

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work proposes a dataset (LBox-Open) containing 150k Korean legal precedents, and proposes two classification tasks, two legal judgment prediction tasks, and a summary generation task based on part of the data in this dataset. And a GPT2 model LCUBE-base is pre-trained using three datasets including LBox-Open. Both the model and the data are open source and can be easily loaded and used through HuggingFace.

**Datasets**  
Their proposed dataset is the first large-scale Korean-language legal-orientation dataset that combines scale and diversity. Contains 63k cases sentenced in the past four years and 96k cases from the first and second level courts in the fact-checking stage, of which a total of 80k are from LAW OPEN DATA searches and 70k are from their own database, covering a total of 1,160 case categories. They have customized a set of data engines to normalize the data format, and uniformly process the data in the original format of pdf and pictures into the format of text.

**Tasks**  
Based on a 150k formatted dataset, they propose 5 tasks. All are based on actual business, and four of them are classification tasks, (1). Predict the type of cases based on facts, (2). Predict violations of laws  based on facts, (3). Predict fines, prison months, and labor imprisonment months based on facts. (4) Predict claims acceptance based on facts and gist of claim. One is the generation task, which is to predict summaries based on ruling and reasoning section.

**Model**  
Based on the three datasets of Wiki, Modu and LBox-Open, they pre-trained a GPT-2 with 124M parameters and compared it with KoGPT-2-base and MT5-small on the above five tasks, providing a task baseline.
strengths: 1. The dataset covers enough case categories, and the data structure is clear and easy to use.
2. The proposed tasks have real business value and solve real business problems.
3. This work has a certain role in promoting the application of AI technology in the legal direction.
weaknesses: 1. I feel a little weak in the experimental part. It is recommended to add a pre-training of the encoder-decoder language model for comparison.
2. On the summary task, the reason why LCUBE is significantly lower than other models can be further analyzed, or some other evaluation metrics of the generation tasks or even some artificial evaluation metrics can be introduced.
correctness: basically correct.
clarity: There are some minor flaws, such as a spelling error in the second line of Section 4.1(fine-tnue), and a spelling error in the Fine-tuning section of github's README(pytyon).
relation to prior work: Yes, in Section 3.2.
additional feedback: Good datasets and tasks, it would be better if the experimental part can be enriched, such as using more structured models, trying large models, or analyzing the phenomena shown in the experimental results in more depth.
review_title: A nice job in the direction of Korean legal Benchmark

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary and contributions: The authors propose a new legal corpus for the Korean language, with a total of 5 associated tasks: two classification tasks, two legal judgment prediction, and one summarization task. They furthermore release a new language model based on GPT-2 architecture but for Korean legal language.  
strengths: The corpus released as well as the processing into the various tasks is a new addition and composed of marginally more data than the previous dataset for Korean law from AI-hub, ranging from 11K to 150K data points. The new language model for Korean legal language is also of practical interest. 
weaknesses: The whole pipeline, from the preprocessing to the evaluation, lacks rigors and makes me doubt the pertinence of the contribution. More specifically:
- The extraction from raw data is produced by ML, with an interconnection of systems and some manually labelled pages. There is no mention of how well each of these systems perform, and no verification on how well they perform together (for example, verification on a 100 randomly selected examples that the extraction was correct). It would also be good to have information such as the threshold for manual labelling, as well as the reasoning behind the choice of that particular threshold.
- For the tasks, only a limited range of examples are included ("100 most frequent case categories", "46 frequent case categories", etc). No justification for these choices are provided; these limitations are not needed since all tasks are modelled as a language modelling task which can be trained with any number of categories; the limitations diminishes greatly the interest for the dataset outside academic research; the task doesn't reflect the performance it would have would it be used in practice.
- The language model LCUBE-base is evaluated and compare to KoGPT2-base on the new tasks, which are part of the corpus used to train LCUBE-base. Given this, it would be surprising if LCUBE wouldn't outperform KoGPT2-base. While limited, an already better evaluation would be to evaluate on the dataset of AI-hub which should be a legal task but from a different corpus.
- One of the argument for this paper is that the dataset from AI-hub is limited in data given that "1.5 million Korean precedents are generated per year". However, the dataset provided is composed of 150K precedents, which doesn't even amount to one year of Korean precedent. The question naturally arises as to why not have a bigger corpus if the data is here and manual labelling is limited in the creation process?
- There is no analysis provided which would help underline the performance of the system and how it could be used. 
correctness: As stated above, several choices in the construction of the dataset are dubious and, if not changed, need strong justifications. Furthermore there are several claims in the paper that are made that should have a citation as they are not common knowledge, ex: "Korea has one of the largest legal industries in the world", or "In the real world, automatic categorization of legal documents or legal questions are often important for various downstream tasks.".
clarity: Correction are needed as mistakes are often found (ex "We pre-trained GPT-2 from the scratch"), the abstract and introduction are dense and not very clear, and rewriting them would be beneficial, but the rest of the paper is clear.
relation to prior work: It is clearly discussed how the work differs from the corpus of AI-hub, but the paper would benefit from a section discussing Korean legal ML or even larger legal ML in order to better establish the relevance of the various tasks.
additional feedback: At the minimum, the work to be done should be 
- A solid rereading for errors
- A better justification of the limitation for the task
- More transparency in the creation process
- Better evaluation of the language model
- Better framing of the ethical problem
- A related work section that justifies more the various tasks

Optimally, the tasks would be re-run without the limitations and with more analysis provided, for example on how the system perform on low-frequency classes.
review_title: LBOX - An interesting proposition but lacking rigor

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors present a multi-task legal service benchmark in Korean for language understanding measures over judgment prediction and many other tasks. The authors also build baselines using existing mono-lingual models as well as multi-lingual models and show enough headroom for future research. The main contribution is that the authors contribute a large open-source LBOX OPEN dataset in Korean for evaluating language understanding for legal documents. This is valuable given that legal documents are hard to harvest and annotate by humans. It also contributes to multi-lingual studies over language understanding for legal documents.
strengths: - The main contribution is that the authors contribute a large open-source LBOX OPEN dataset in Korean for evaluating language understanding for legal documents. 
- This is valuable given that legal documents are hard to harvest and annotate by humans. 
- It also contributes to multi-lingual studies over language understanding for legal documents.
weaknesses: - Baselines still suggest that a multi-lingual model such as MT5 performs better than in-distribution pretrained LCUBE-base. It is worth exploring even large models such as GPT-2 or GPT-3, if not MT5-base or large, to benchmark existing multi-lingual models with this benchmark. 

- It would be interesting to see how an English pre-trained language model performs by directly asking it to transfer to do legal documents understanding in another language (assuming we have some hacks in tokenization).

- It would be interesting to see more error analyses over these baselines. For instance, what are the cases that MT5 gets wrong in terms of CASE NAME classification? Is this because the document is too long? Or it contain complex descriptions in Korean?


correctness: The claims made in the submission seem to be correct to me.
clarity: This paper is well written.
relation to prior work: Related works are well discussed.
additional feedback: No.
review_title: interesting benchmark with solid contribution on building AI systems around legal services

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: Large-scale corpora, benchmark datasets, and language models in the legal domain are lacking. This paper remedies this by introducing this for Korean Law. It introduces Korean legal AI benchmark datasets that consist of six datasets: one large-scale legal precedent corpus, two classification tasks, two legal judgement prediction tasks, and one summarization task. In addition, the a large-scale Korean legal language model is released. As part of the (pre)processing, the authors processed document images and PDFs. The pipeline is presented and explained.
strengths: The paper contribution consists of the introduction of a large-scale corpora, benchmark datasets, and language models in the domain of Korean Law. Legal datasets are scarce, particularly in different languages than the English language. This dataset offers an important resource for empirical legal researchers and computational researchers who wish to analyze Korean case law. The six datasets that are introduced (and benchmarked) have the potential to provide for a significant breakthrough. The same applies to the Korean legal language model that is released. To my knowledge, it is the first of its sort. It was quite the endeavor to compose the datasets and the model, which makes the construction a significant effort that is unlikely to be done by others.

One could argue that a Korean legal dataset has limited relevance to the broader research community. By that standard, the vast majority of non-English datasets can be excluded. As the authors point out, there is a knowledge gap with respect to NLP models (and data) in languages that are not English.

The dataset is made available on Github. Although I have not ran the code myself, the documentation seems sufficiently clear and transparent in order to access the datasets.
weaknesses: The authors speak about precedents, but it does not become entirely clear what is meant with precedents. As the authors rightfully point out, precedents are cases from the past that have produced a rule that can be used to decide subsequent cases. Many cases are not considered precedents, although on a generic level every court decision can be seen as a precedent. It seems that the author follow the latter approach, essentially equating precedents with court cases. This is fine, but it would be good if the authors would explain this. This is a minor comment.

The paper speaks about 6k precedents that were collected as part of approximately 1.5 million Korean precedents in the total population. Maybe I missed it, but I did not encounter possible explanations for the discrepancy. For instance, could selection bias be involved? It remains difficult to determine the generalizability of the dataset.

I am not sure what is measured in the 'prediction' part. What is predicted, is the outcome of the decision based on the text that the court produced. This is more likely to be a prediction of how judges motivate their decisions in case of a certain outcome rather than predicting outcomes based on 'the facts'. The results in the 'LCUBE-base + reason (oracle)' model are surprising. As previous research shows, one would expect that the outcomes of court cases can be predicted with high levels of accuracy when including the reasoning of judges. Or did I not correctly understand what was included in the 'reason (oracle)' here?


correctness: The evaluation methods and experiments seem sound.
clarity: Yes.
relation to prior work: The paper would benefit from including more references to studies that aim to predict outcomes of court decisions.
additional feedback: The paper contains some typo's.
review_title: Review - A Multi-Task Benchmark for Korean Legal Language Understanding and Judgement Prediction

Meta-review: metareview: This paper presents a large corpus of Korean legal documents, paired with labels corresponding to two classification tasks, two legal judgment prediction tasks, and one summarization task. Reviewers praised the uniqueness of the new resource.

However, there is some criticism of the results section, with several reviewers bringing up possible comparisons to other models and wondering whether the performance of this model has been analyzed thoroughly.  The authors argue in particular that mT5 and KoGPT-2 can benefit from continued pre-training this corpus, which seems satisfactory, although there is the question raised by 7fpy that this is essentially pre-training on the test tasks and may be too generous when assessing the performance on other legal-domain tasks that arise down the road.  Reviewer 7fpy brings up some valid critiques of the dataset construction: several points about filtering were unclear, as was the strength of the automatic pipeline used to construct the dataset.

There are a few ethical concerns raised by the ethics reviewer, which I believe are serious. Point [1] is resolved. Point [2] about misuse seems valid and harder to argue against.  The data does have value for use in the kind of more benign studies the authors report in the response there. But its very existence could encourage the construction of certain kinds of automated tools for legal judgment prediction, which has been a hotly-debated issue in the NLP community before. Ultimately I will defer to the ethics reviewer on this one, who seems satisfied.

Overall this paper seems like a well-done effort. The main question is whether its utility (modulated by issues with the results) outweighs the risks of putting it out there. Reviewers seem to lean positive on this and I would lean positive as well.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept