dataesr/scientific-paragraphs-categorization huggingface dataset:
Experimental Results Table  3  shows the experimental result of the translation system with and without the correction module. With a relatively small dataset, the translation module itself can produce a translation that has about 45% of exact match accuracy with the reference translation  and 94% of LCNS. The correction module can further improve the translation quality to 58% of exact match accuracy and 97% of LCNS. We can have three observations from these experimental results: • The translation module alone can produce a translation with fairly high quality. • The correction module can further improve the translation quality. The improvement is not that significant given the small training dataset, but it is significant enough to show that the correction module has the potential to improve the translation quality.
• The translation quality can be further improved if we have a larger training dataset. While the Exact Match and LCNS metrics show us the overall performance and improvement of the system, it's hard to imagine how good the system really is and what errors the correction module can detect and correct. Therefore, in Section 4. 3, we analyze the specific cases that we observed in our experiment. Table  4  shows some translation examples for the translation system with and without the correction module. The reference translation is provided in the first column. The translation generated by the translation module is in the second column and the translation generated by the translation module with the correction module is in the third column. We can see that the system can extract the key information from the input sentence and generate a translation with fairly high quality.
Although the predicates in the target logical form do not appear in the input sentence, the system can produce a correct translation by reusing the existing predicates in the training dataset. In the version with solely the translation module, the system sometimes misspells the predicates and variable names in the logical formula. The correction module can correct some of these errors. In some cases, the correction module can also correct the case of the predicate names and variable names, which is important for the correctness of the logical formula. In the case of wrong date translation, the correction module based on noised generation can only reformat the date but can not assure the correctness of the date.
Discussions From the experimental results, we can see that the translation system can achieve fairly high quality with a small amount of training data. However, the system still has some limitations. First, in our current dataset, the number of predicates and variables is limited. Therefore, the translation system can not generate a logical formula containing predicates and variables that are very different from the predicates and variables in the training dataset. Second, the translation system is not robust in recognizing the dates. In the third output in Table  4 , the translation system translates the date "1995 year 11 month 13 day" to "2000 year 11 month 13 day". This result should be due to the lack of date recognition skill in the model and overfit to the training dataset. The correction module can not deal with this kind of error.
Conclusions In this work, we propose a translation system that translates natural language text into logical formulas. We use PROLEG syntax as our target language. As an investigation step, we hire experts to create a small translation dataset from natural language to logical formulas. To improve the translation performance, we propose the multi-step framework by appending the correction module to deal with translationś errors. The correction module is constructed by learning a reverse function from the error generator. The experimental results show that this design can improve the translation quality. The translation system with the correction module can achieve 58% of exact match accuracy and 97% of LCNS in the current small dataset. This work is a promising first step toward automatically translating natural language into logical formulas.
Data Description and Preprocessing The dataset manually examined by  [1]  consisted only of 149 cases from 2017. We extended this to 12,102 cases spanning 2001 to 2018 that included clearly separated sections described by  [17] : facts (rich with real-life situations described by laymen) and legal analysis (rich with legal terminology). Clear section boundaries are necessary because, unlike works from Section 2 and following  [10] , we do not carry out topic modeling on the entire document but only on the facts section so that our topics contain as little legal terminology as possible and more real-life situations. As in  [1] 's work, we retained cases citing articles 1854, 1864 or 1910 from the Civil Code of Québec in which the tenant is the applicant, thus amounting to 1,381 cases. Since each case can contain several litigation factors, topic modeling is done at the paragraph level.
Our resulting dataset of 34,685 paragraphs is processed in a standardized fashion with a SpaCy tokenizer: We remove dates, monies, digits, symbols, French-and law-specific stopwords, then filter tokens with specific part-of-speech tags, lemmatize and lowercase them before merging bigram collocations (e. g. hot water). We remove paragraphs with less than 5 terms, yielding 26,815 paragraphs.
Latent Dirichlet Allocation (LDA) LDA is a generative stochastic model that aims at recreating the original corpus through a pseudo-corpus generation. For a preset number of topics, it generates a collection of pseudo-paragraphs whose word and topic distributions approximate as closely as possible those of the real dataset. Paragraphs are considered as random mixtures over latent topics, and topics as distributions over words. Dir(α) is the Dirichlet distribution of topics over paragraphs while Dir(β ) is the Dirichlet one of words over topics. Dir(α) is the prior for multinomial topic distribution θ p for paragraph p while Dir(β ) is the prior for multinomial word distribution φ k for topic k. As shown on Figure  1 , at position i of pseudo-paragraph p of pseudo-corpus C, word w p,i is defined by both θ p that defines the Table  1.
Besides these works based on sentence-extraction, more and more sophisticated models were developed for abstractive summarization by  [11, 12, 13 ], but such approaches were mostly applied to news datasets. To the best of our knowledge, the only benchmark available for abstractive summarization of legal documents is BigPatent  [14] , though this corpus consists of patents and not court judgments as in the aforementioned extractive summarization tasks. Several reasons may explain the lack of accessible benchmarks for legal judgment abstractive summarization: • Summaries written by legal experts are prohibitively costly to obtain, due to the length and complexity of decisions, plus the scarcity of legal practitioners who must be familiar with the target legal area ; • Since decisions are significantly longer than documents in generic NLP corpora, they can be hard to process for models whose maximum input length is limited despite the emergence of transformer-based  [15]  models such as  [16, 17]  that can process longer documents but at a high computing cost.
Data and Task Description We extracted from the JusticeBot database and from CanLII a total of 156 instances. Although the dataset is small, it reflects diverse real-life issues faced by users. Each instance has 4 pieces of text: • the question Q and the answer A from the JusticeBot interface ; • the main text of a decision D that we extracted with heuristics from the HTML page from CanLII (metadata are removed) ; • the summary S about D provided by an annotator to the JusticeBot user. The task consists in mapping Q, A and D to the target summary S. The mean/median number of tokens for Q, A, D and S amount to 13/13, 1/1, 1030/745 and 44/41, respectively. Because of the dataset size, we will run our experiments with 10 folds, each fold having a train/validation/test split ratio of approximately 80:10:10.
Such observations are consistent with those drawn by domain experts in the summarization task conducted by  [6]  ; • although BART architecture has a larger maximum input sequence length (1024 tokens) with respect to commonly used transformer models (512 tokens for BERT  [25] ), reverting the paragraphs order of a court case allows minimizing the risk that important information located towards the end of the decision is not included in the input sequence of the model. Each aforementioned combination of text is provided as input to both VanBART and FPTBART. For each fold, the model is fine-tuned with the Adam optimizer. Given the small dataset size, the batch size is 1. In order to smooth out the optimization of crossentropy loss, we apply an initial learning rate of 10 -4 with a scheduler that halves it at the end of each training epoch if the ROUGE-1 score does not improve on the validation set.
Results and Discussion For each model and each combination of text inputs, Table  1  gives the average scores across 10 folds for BLEU  [26] , ROUGE-1, ROUGE-2, ROUGE-L  [27]  along with stan-dard deviation. Overall, for a given input combination, FPTBART delivers a higher performance with respect to VanBART. Such improvement is consistent with the fact that unsupervised pretraining of a transformer model helps for tasks in specialized domain as shown by  [28]. Considering FPTBART results, the combination of question, answer and decision (QAD and QADr) seems to perform best in terms of ROUGE and BLEU as they contain more information and slightly outperform the combination of question and decision only (QD). Still, as it is hard to appreciate such syntax-based measures given the dataset size and the nature of our experiment, we retained output summaries from settings QD, QAD and QADr for manual evaluation.
Qualitative Analysis Upon manual examination, some summaries appeared easier to generate than others, especially those that consisted in a single sentence and/or are related to rent arrears (the most frequent issue in our dataset and in real life), although models also struggle with time duration. This is shown in Example (a) in Figure  3  where all candidate summaries accurately describe the tenant as being late in payment, with QADr adding 3 months of delay. For more complex cases, on the contrary, output summaries are less convincing as models tend to repeat phrases used in somewhat similar cases but without properly addressing the instance at hand. Example (b) provides a good illustration with the blue highlighted segments that contradicts the reference summary. Moreover, the last candidate (QADr), although being consistent with the input triplet and target summary, is making an extrapolation about article 1943 C.
Conclusion We implemented a novel conditional text generation task aimed at making summaries of court decisions for laymen by employing a state-of-the-art generative model. In addition to using automatic metrics (AM), we designed domain-specific manual criteria (MC) for sensible human evaluation and to assess to what extent generated summaries meet Jus-ticeBot's user requirements. Overall, the best performance in terms of AM and MC was achieved by a further-pretrained BARThez that gets a question-answer-decision triplet as input, but a closer examination of output summaries reveals large room for improvement. As future work, we plan to: obtain a much larger dataset that covers more cases ; find methods to guide and control summary generation at a finer grain in order to incorporate legal reasoning ; examine how transformer attention weights handle each component of triplet input.
Dataset We obtained a dataset of 715 unique case briefs by scraping a publicly available Case Brief Summary database. 3 We used an extensive battery of regular expressions to segment the retrieved briefs into individual sections corresponding to the key argument element types. While there were over 100 unique section names we identified the six main types (see Section 2) to which we could map many of the different variations (e. g. , all of "Legal Issue", "Issues", and "Issue" map to a single category). We applied a specialized legal case sentence boundary detection system to segment the sections into 9,924 sentences. [27]  Figure  3  shows the distribution of the sentences in terms of the key argument element types from the perspective of their overall counts as well as their distribution over the individual case briefs.
Model and Training As a baseline we use a simple model that randomly predicts the labels based on their frequency in the dataset. We use the standard implementation provided by the sklearn framework, with "stratified" sampling. 4 As the main model we employ RoBERTa (a robustly optimized BERT pretraining approach) developed by Liu. [17]  The model was chosen due to its high performance and simplicity to train. While higher performance might be achieved by more recent models, the RoBERTa model suffices for the purpose of proving the feasibility of the key components of the framework. Out of the available.
Testing H1: Warning about an Incorrectly Categorized Argument Element When interacting with an Intermediate user (NIH Level 3) CABINET issues a warning if a piece of text identified by the user as a key argument element has been (highly likely) assigned with to an incorrect section (see Section 4). Hence, the input is a short text together with a label assigned by the user. The system either issues a warning about the assignment being likely incorrect or abstains. We transform the dataset by creating text-label pairs between each sentence and all the labels. Since there are 1,440 sentences in the test set and 6 unique labels, there are 8,640 such pairs. For each pair, we retrieve the probability distribution the model assigns over the possible labels. If the value for a given pair is below a static threshold (we experiment with 0.
First, the design of the experiments only takes into account passages of text that have been selected for inclusion in the case briefs by legal experts. However, the user may occasionally make mistakes in their selections. This is particularly true for users at the Intermediary proficiency level (NIH level 3). Second, we do not address the challenge of assessing the current level of proficiency of the user. Third, the functionality of the system at the Fundamental Awareness (NIH Level 1) proficiency level requires a limited but highly curated dataset of annotated cases with detailed feedback addressing common misconceptions-a resource we have not yet created. Fourth, we did not conduct a feasibility study of the highlighting functionality at the Expert proficiency level (NIH Level 5). Fifth, and most importantly, we have not conducted a pilot user study to tentatively gauge the expected improvements in learning outcomes.
Introduction and Background In Common Law legal systems, cases are resolved by reference to prior cases involving similar claims. Datasets of such cases, and their developers' formal commitments and assumptions, have been important to AI & Law research. We present a dataset of case texts translated into predicate logic using a publicly available natural language understanding system. Several large datasets collect cases in their original language  [1, 2] , and researchers using large-scale machine-learning techniques have recently developed legal retrieval and reasoning systems that operate over raw text  [3]. But many AI legal reasoning systems require formal machine-interpretable representations, with which researchers have had to annotate their cases. The most widely-used formalism is factors  [4] , legally-salient principles relevant to finding case outcomes and that favor a party.
The Illinois Intentional Tort Qualitative Dataset Illinois was chosen as the dataset's jurisdiction because our institution is located there. CNLU cannot currently handle statutes' linguistic complexity and legalistic formalism, so we sought pure common-law doctrines, where judicial opinions express rules in plain English. We included the Tort doctrines of Trespass, Assault, Battery, and Self-Defense. We found cases using WestLaw and LexisNexis and traced their cited and citing cases. Cases were excluded if unreasoned, overturned, decided before 1870, or unrelated to prior or subsequent cases. Collected cases were organized by doctrine and annotated with decision year, court, and case reporter metadata. Case facts and conclusions were manually identified and stored as a string argument to a metadata fact.
In eleven cases, appellate courts laid out alternate set of facts left ambiguous by a lower court and identified the corresponding legal conclusions. Each of these was converted into two dataset cases, for each set of facts and conclusion. The dataset comprises 88 cases illustrating 112 distinct tort claims. These include 17 assault cases (12 positive cases where the court found an assault had occurred, 5 negative cases where the legal standard was not met), 40 battery cases (30 positive, 10 negative), 43 trespass cases (29 positive, 14 negative), and 12 self-defense cases (5 positive, 7 negative). Positive cases outnumber negatives because they are more likely to be published and later relied upon. Judges' descriptions of case facts often include run-on sentences, long lists and descriptions, and asides, so case texts were simplified for CNLU (Figure  2 ).
Here we report performance for ARPG with a strict truth test and PAPR with a partial truth test, our floor and ceiling performances in  [21]. Finally, we tested our systems only on Trespass, Assault, and Battery; we are still studying modeling affirmative defenses like Self-Defense. We report two baselines using off-the-shelf ML techniques. LegalBERT is a BERT model specialized on legal texts  [22]. It was tested as a multiple-choice system. We created multiple-choice cases by negating conclusions and reversing party roles. GPT-J is an open-source model based on OpenAI's GPT-3  [23]. GPT-J was fine-tuned and tested on our dataset using holdout and 5-ply cross-fold validation. To test GPT-J, we prompted it with the simplified case facts and had it generate six text completions, which we examined for a simple expression of the conclusion.
Discussion and Limitations We discuss implications the analysis reveals about the dataset; methods are discussed in  [21]. A greater number of positive than negative cases, as well as cases where the defendant is the one accused of tortious conduct, may lead statistical methods to accuse the defendant and generally perform well. Indeed, statistical methods outperformed ARPG in positive but not negative cases. To assess a reasoning technique's performance, special attention should be paid to negative cases and those where someone other than the defendant is the accused. The limitations of CNLU and of the process by which commonsense NF rules were generated must be acknowledged. The dataset does not yet reach the goal of being generated by feeding raw legal text into a language understanding system, because no system can reliably both handle the complexity of legal texts and generate accurate symbolic representations from them.
CNLU features three limitations, each of which is an area of active research. First, CNLU still relies on a human simplifying the original text: the complexity of surface forms CNLU can handle has progressed, but it cannot yet handle arbitrarily complex English input. A stopgap solution may be to train a large language model to simplify texts to CNLU's level. Second, for this dataset CNLU's choice sets were manually selected to ensure semantic fidelity to the text. CNLU can automatically select choice sets quite well  [14, 24] ; we hand-selected because our goal was to create an accurate dataset, not to evaluate CNLU. Third, something like NFs are necessary to express what a text means, not just what it literally says. Common-sense reasoning is a persistent problem in AI research. For now, the options are to create generally-applicable rules, or to accept that facts obvious to humans remain unknown.
Dataset The dataset includes 1704 Judgements issued by the Court of Justice of the European Union between 1954 and 2020. All judgements are in English and freely available from the official portal of the European Union Eur-lex. Content-wise, the judgments concern the free movement of goods and the free movement of persons, both an ideal test bed. On the one hand, the Court has fashioned the fundamental principles of European Union law and developed its central doctrines in those areas, which makes them ideal for the prediction of legal importance (doctrinal outcome). On the other hand, with the development of fundamental principles, it became relevant whether the Court left the key practical decisions to the courts and the legislators of the Member States (deference). Legal experts (human coders) labelling each judgment specified whether the judgment was legally important (Doctrinal Outcome or DOCOUT) and whether the Court deferred the final decision to the national courts or legislators (Deference or DEF).
The data is divided in two datasets: the first contains the full judgements and their relative predicted label. The second contains single paragraphs of the judgments, each with assigned label predicted for their judgement. Compared to similar researches our dataset is small: Wei et al. [8]  sampled from a dataset composed by millions of judgments; Xiao et al. [9] , in 2018, used a dataset composed by 2. 6 million criminal cases published by the Supreme People's Court of China. Small training data is commonly known to result in poor performance, particularly in Deep Learning  [10]. However, it is time expensive to produce hand coded training sets.
We restricted the size of the vocabulary for the word embedding to 30000 words. The datasets were randomly divided in a training set and a test set with 85% and 15% of the data. The training set was furthermore randomly split into 4 cross validation folds for the model training. Finally, the performance of the models are computed on the test set. We trained our models until convergence, using an early stopping criterion that monitored the F1 score, using the Adam optimization algorithm and Binary Cross Entropy as loss functions. This is a good strategy which prevents over-fitting and saves some computational time.
This might indicate some under-fitting. The overall performance of the models is weak due to a number of difficulties: small training dataset, substantial class imbalance, and the fact that predicting doctrinal outcome is likely to require extensive knowledge of complex legal concepts and legal doctrine. The findings are nonetheless important and telling. First, LSTM appears to perform better than CNN on all tasks. Future research could push the results forward by increasing the size of the training data (which implies time consuming and expensive hand coding of the data). The performance of deep learning models is known to increase with the size of the training data  [10]. Alternatively, researchers could explore more complex models: from multiple LSTMs in sequence, to encoder-decoder architectures, and more recent BERT-based models  [16].
Conclusions The article investigated how much machine learning could contribute to the legal analysis of judicial decisions by predicting two legally interesting and demanding characteristics: legal importance (doctrinal outcome) and deference. It trained a set of classifiers based on word embedding, LSTM, and CNN on a dataset of manually labelled judgments of the European Court of Justice. The tasks proved difficult and performance were weak, with LSTM performing better than CNN. Further analysis and experimentation would be required to understand the significance of these results. These include: developing more sophisticated models, incorporating more hand-coded judgements, and finding ways to deal with long text sequences. This work can be viewed as a starting point for studying the impact of text classification and the potential of deep learning models in very specific NLP fields, such as the legal domain.