dataesr/scientific-paragraphs-categorization huggingface dataset:
Introduction Statutory reasoning is the task of reasoning about legal cases with legal statutes. It is generally contrasted with case-based reasoning, and is complementary to it in legal systems based on common law  (Lawsky, 2017). The StAtutory Reasoning Assessment dataset (SARA) is a benchmark for statutory reasoning for US federal tax law  (Holzenberger et al. , 2020). In some settings, laws and regulations can be encoded into first-order logic programs to reason about cases  (El Hamdani et al. , 2021; Merigoux et al. , 2021a; Bench-Capon et al. , 1987). In  Holzenberger et al. (2020) , SARA statutes and cases were manually translated into Prolog code, solving statutory reasoning in that context, in a process depicted in Figure  1. Here, we take the standpoint advocated by  Merigoux et al.
Prolog engine In this work, we develop a form of Information Extraction (IE) for legal cases  (Filtz et al. , 2020) , corresponding to the task highlighted on the right of Figure  1. We release a new version of the SARA dataset, with updated annotations for the Prolog KBs, and establish baseline results in extracting those KBs from case descriptions. This allows us to answer the following: • What representation of Knowledge Base Elements (KBEs) in Prolog is appropriate for IE? • How well can IE models perform on SARA? • How does performance in IE translate to performance in statutory reasoning? We report improved scores on statutory reasoning, showing the benefit of a structured Prolog representation for statutes. We further show an intuitive correlation in IE performance with performance in statutory reasoning.
, 2022). IE has also been attempted on regulatory texts:  Chalkidis et al. (2019)  released a dataset of legal documents produced by the EU annotated with a set of legal terms. There are datasets for classifying paragraphs in policies  (Bartolini et al. , 2004) , extracting subjectobject relationships  (Alohaly et al. , 2018) , and identifying cross-references  (Boella et al. , 2019). Closer to this paper, IE has been applied to case law. Yao et al. (2022)  introduce a dataset of Chinese court cases annotated with a custom event ontology. Navas-Loro and Santos (2018)  offer a review of event extraction in the legal domain, both annotation schemas and computational models, as well as annotations of European Court of Justice cases. Filtz et al. (2020)  provide a corpus of events based on the decisions of the European Court of Human Rights, and compare statistical and rulebased models for event extraction.
Universal Decompositional Semantics  (White et al. , 2020)  provides multiple layers of annotations, separating syntactic and semantic information, and decompositional semantics. There are semantic representations that aim to reflect the full content of language, such as Episodic Logic  (Kim et al. , 2021; Schubert et al. , 2010). Closer to this work, the PropBank dataset  (Palmer et al. , 2005; O'Gorman et al. , 2018)  adds predicate-argument relations on top of the Penn Treebank parse trees  (Marcus et al. , 1993). This effort has been extended to more languages  (Moeller et al. , 2020; Anwar et al. , 2016; Wu and Palmer, 2015). The structures considered here are represented using Neo-Davidsonian semantics  (Castañeda, 1967; Davidson, 1967; Parsons, 1990). In particular, they only capture the content relevant for interpreting the SARA statutes.
Previous work on semantic representations for the legal domain, including the tax-law domain, have made use of ad-hoc ontologies, depending on the application domain  (Bench-Capon et al. , 1987; Sherman, 1987; Merigoux et al. , 2021a). Task-oriented representations When performing semantic parsing for a specific task, it is possible to sacrifice expressivity for performance  (Kollar et al. , 2018). Zelle and Mooney (1996)  introduce CHILL, a shift-reduce parser producing queries executable against the Geoquery database. With the same benchmark dataset,  Zettlemoyer and Collins (2005)  induce a grammar and a log-linear model to map questions to queries. Similarly,  Berant et al. (2013)  answer questions stated in natural language to a KB. They translate the question into a query that can be executed against the KB.
Beltagy et al. (2014)  use semantic parsing for the tasks of Recognizing Textual Entailment and Semantic Textual Similarity. Zhang et al. (2019)  and  Weir et al. (2020)  generate SQL queries from natural language questions. Here too, compositionality can play a role  (Pasupat and Liang, 2015). Campagna et al. (2019)  focus on parsing language for virtual assistant commands. Goldwasser et al. (2011)  introduce a method to train a semantic parser with selfsupervision, using a model's own high-confidence predictions on the Geoquery dataset as training examples. Liang et al. (2011)  offer an alternate approach, treating the semantic parse as a latent variable. We are not aware of any prior work on legal-domain task-oriented semantic parsing.
Annotation of SARA In  Holzenberger et al. (2020)  and again in Holzenberger and Van Durme (2021), the semantics of each SARA case have been translated into a set of KBEs. These annotations, based on a custom ontology in Prolog, effectively make it possible to solve SARA with a Prolog program that reflects the semantics of the statutes. However, with IE in mind, the ontology has several design flaws. We will focus on two of them. First, the events are represented using strings and integers, without clear reference to the text of the case description. Second, the use of the ontology is somewhat inconsistent. We improve over this annotation scheme to enable the training of IE models. 1.
Prolog program We adjust the Prolog program to the changes mentioned above. We implement functions to extract year, month and day from the integer representations of dates. When comparing In 2017, Alice's gross income was $326332. Alice and Bob have been married since Feb 3rd, 2017, and have had the same principal place of abode since 2015. Alice was born March 2nd, 1950 and Bob was born March 3rd, 1955. Alice and Bob file separately in 2017. Bob has no gross income that year. Alice takes the standard deduction. 1 income_("gross income") income_(span("income", 23, 28)) 2 start_("gross income", "2017-12-31") start_(span("income", 23, 28), span(20170101, 3, 6)) 3 agent_("gross income", "Alice") agent_(span("income", 23, 28), span("Alice", 9, 13)) 4 amount_("gross income", 326332) amount_(span("income", 23, 28), span(326332, 35, 40)) 5 marriage_("married") marriage_(span("married", 67, 73)) 6 agent_("married", "Alice") agent_(span("married", 67, 73), span("Alice", 43, 47)) 7 agent_("married", "Bob") agent_(span("married", 67, 73), span("Bob", 53, 55)) 8 start_("married", "2017-02-03") start_(span("married", 67, 73), span(20170203, 81, 93)) 9 residence_("principal place of abode") residence_(span("abode", 137, 141)) 10 agent_("principal place of abode", "Alice") agent_(span("abode", 137, 141), span("Alice", 43, 47)) 11 agent_("principal place of abode", "Bob") agent_(span("abode", 137, 141), span("Bob", 53, 55)) 12 patient_("principal place of abode", "the same principal place of abode") patient_(span("abode", 137, 141), span("place", 128, 132)) 13 start_("principal place of abode", "2015-01-01") start_(span("abode", 137, 141), span(20150101, 149, 152)) 14 birth_("Alice was born") birth_(span("born", 165, 168)) 15 agent_("Alice was born", "Alice") agent_(span("born", 165, 168), span("Alice", 155, 159)) 16 start_("Alice was born", "1950-03-02") start_(span("born", 165, 168), span  (19500302, 170, 184) ) 17 end_("Alice was born", "1950-03-02") [REMOVED] 18 birth_("Bob was born") birth_(span("born", 198, 201)) 19 agent_("Bob was born", "Bob") agent_(span  ("born", 198, 201), span("Bob", 190, 192) ) 20 start_("Bob was born", "1955-03-03") start_(span  ("born", 198, 201), span(19550303, 203, 217) ) 21 end_("Bob was born", "1955-03-03") [REMOVED] events, we compare value, start and end indices.
Data pre-processing First, we run Prolog queries against each case to produce an exhaustive inventory of events and arguments. For example, the query ?-income_(X). returns all events of type income_, and the query ?-agent_(X, Y). returns all X, Y pairs such that entity Y is the agent of event X. We thus produce a set of cases containing only KBEs, to avoid the need to extract the rules shown in Appendix A. This dataset we refer to as the full dataset. Some spans cannot be extracted using the IE framework described in Section 4. First, it cannot deal with overlapping spans. Out of 2 overlapping spans we keep the one that appears earliest in the case description. Second, some values cannot be inferred from their anchor. For example, some sentences describe repeating events, without explicitly mentioning all the dates of occurrence.
This makes it impossible to extract those unmentioned dates. Appendix A describes one such event. We produce the partial dataset by removing all unextractable spans and corresponding KBEs. Table 1 shows statistics on how many KBEs were removed in the process. The number of KBEs rejected can seem large in comparison to those kept. However, a fraction of the cases contains most rejected KBEs. Ranking the cases by number of KBEs rejected, the top 10% represent 96%, 98% and 71% of the rejected KBEs in train, dev and test, respectively. Figure  4  in Appendix G shows statistics on the partial dataset. At training time, we use the partial dataset. At test time, we report IE scores against both the partial (Table  2 ) and full datasets (Table  4 ). To clarify the difference: the partial dataset contains only those KBEs from the full dataset that our framework is designed to handle.
Both contain the same SARA cases. Note that this filtering only mildly affects the overall task of statutory reasoning, as can be seen in Table  3  The SARA dataset only provides a train/test split. For the purposes of early stopping and hyperparameter search, we split up the train set into a train and dev set, the same way that the train/test split was created in  Holzenberger et al. (2020). We pair binary cases by the part of the statutes that they test: e. g. s2_b_1_pos and s2_b_1_neg are grouped together. We randomly select 20% of these groups for the dev set. We also randomly pick 20% of numerical cases to be used in the dev set. The result is a train set with 206 cases and a dev set with 50 cases.
Results Information Extraction We use the AllenNLP codebase  (Gardner et al. , 2018)  version 2. 7. 0 to code and train models. 6  We use stochastic gradient descent with early stopping on the dev set. Performance of a given model is measured as its F1 score, averaged across cases. We count a KBE predicted by the model as correct if it appears exactly in the gold KB. The model receives no credit for a slightly inaccurate span or label. We use Optuna  (Akiba et al. , 2019)  to search for the best set of hyperparameters. Appendix F reports the range of hyperparameters explored and best hyperparameters found. We measure performance for a given set of hyperparameters as the mean F1 score across 3 different runs, with 3 different random seeds. This is to mitigate variability from random initializations.
In Table  2 , we report precision, recall and F1 scores, for dev and test sets, on the partial dataset. We report results on the full dataset in Table  4  in Appendix D. We compare 5 different pre-trained encoders (used to map X to H in Section 4): • LEGALBERT: the legal-domain encoder of  Holzenberger et al. (2020) , trained on a subset of the case. law corpus (caselaw), starting from bert-base-cased, • LEGALBERT': an alternative trained by  Zheng et al. (2021)  on more data, • BERT: BERT-base-cased  (Devlin et al. , 2019) , which has the same architecture and training objective as both LEGALBERTs, but was trained on general-domain English, • ROBERTA: RoBERTa-base  (Liu et al. , 2019)  which was shown to achieve higher performance than BERT on downstream tasks, and • T5: T5-base  (Raffel et al.
Discussion Focusing on the IE scores, LEGALBERT performs best out of all encoders, and T5 performs worst, with large performance gaps between encoders. For T5, only the encoder was used, and presumably, more training data is needed to retrain it as a standalone encoder. Legal-domain pre-training clearly yields performance improvements. As can be seen by comparing LEGALBERT and LEGALBERT', for the SARA dataset, more data does not automatically translate to better performance. As noted in the caption of LEGALBERT 88. 7 ± 0. 6 88. 6 ± 0. 6 88. 1 ± 0. 3 88. 1 ± 0. 2 91. 0 ± 0. 5 87. 0 ± 0. 3 LEGALBERT' 87. 4 ± 1. 0 83. 2 ± 1. 4 84. 4 ± 0. 8 85. 4 ± 0. 9 84. 6 ± 1. 4 82. 7 ± 0. 9 BERT 85. 9 ± 2. 1 75. 1 ± 2. 5 78. 7 ± 1. 6 85. 2 ± 2. 3 77. 0 ± 2. 3 77. 4 ± 1. 8 ROBERTA 84. 5 ± 0. 6 80. 8 ± 1. 5 81.
9 ± 0. 9 86. 4 ± 0. 7 83. 9 ± 1. 6 82. 0 ± 1. 0 T5 73. 2 ± 0. 6 56. 2 ± 2. 4 62. 1 ± 1. 5 71. 6 ± 1. 1 59. 0 ± 1. 3 61. 4 ± 1. 0 EMPTY on statutory reasoning is higher than that. First, with the EMPTY KBs containing no facts, it is generally hard to prove anything at all, so that negative cases tend to succeed thanks to Prolog's negation as failure. Second, since statements about statutes are added automatically to the KB, cases that rely on them have a chance to succeed. There is overall strong Pearson correlation between F1 and statutory reasoning scores: on the test set, 80% for entailment accuracy, and 88% for numerical accuracy. This shows that, for the SARA dataset, improvements in IE translate to improvements in statutory reasoning. This suggests that IE is important for statutory reasoning, and encourages progress on this task.
It would also make it possible for the logical representation to align more closely to the language of the case description. This would yield a better inductive bias for the IE models while not sacrificing expressivity. More generally, progress on IE could be made by picking a more expressive, more general ontology, such that the translation between language and logical form would be more straightforward. But this would require re-writing the Prolog solver, to accommodate the new ontology, and to encode relevant commonsense knowledge. Finally, the use of explicit KBEs introduces interpretability and auditability in the model's output, since it becomes clear what facts are used to perform statutory reasoning. This is generally absent from the direct application of LLMs to statutory reasoning, even if this can be mitigated with prompt engineering  (Blair-Stanek et al.
Conclusion We improve the SARA ontology and re-annotate cases for their semantics. Compared to the previous iteration of the SARA dataset, the KBEs and overall structure are now richer and more complete. We also take a step towards more consistent annotations, and more extractable KBEs. We adapt state-of-the-art IE models to extract KBEs from case descriptions. With empirical results on the SARA dataset, we find that good performance with a strict metric can be reached on this small dataset, and that performance in IE translates directly into performance in statutory reasoning. We see three main directions for future work. First, as stated in Section 5. 1, the current ontology is not flexible enough to both (1) allow for KBEs closely aligned to the case description, and (2) express complex relationships, in particular time relationships.
This is one major limitation that could be addressed, and would likely require overhauling the expert system representing the statutes. Second, we have so far assumed that the statutes were parsed into a logical form ahead of time. A meaningful step would be to augment the approach presented here with the automated extraction of the logical form of statutes, effectively performing semantic parsing. Finally, cases in SARA are not representative of the full scope and diversity of legal cases. Real-world legal cases are generally much longer, the language is denser, and the phrasing is more diverse. We see the choice of an ontology and the annotation of data as two crucial challenges.
Limitations In this paper, we performed experiments on the SARA dataset, a semi-synthetic dataset for US federal tax law. The conclusions drawn about legaldomain IE and about statutory reasoning should take care to understand that this dataset is not representative of the full scope of legal data. span  ("Bob", 282, 284) ). amount_  (span("earned 2017", 286, 291), span(300000, 294, 299) ). start_  (span("earned 2017", 286, 291), span(20170101, 317, 320) ). income_  (span("earned 2018", 286, 291) ). agent_(span  ("earned 2018", 286, 291), span("Bob", 282, 284) ). amount_  (span("earned 2018", 286, 291), span(300000, 294, 299) ). start_  (span("earned 2018", 286, 291), span(20180101, 317, 320) ). income_  (span("earned 2019", 286, 291) ). agent_(span("earned 2019", 286, 291), span("Bob", 282, 284)).
E Error analysis We compare model outputs on 3 binary cases and 3 numerical cases picked at random from the test set. We only mention the errors made by the models. Table  5  shows the answers produced by the Prolog program and the KB on those 6 cases. T A X_C A S E_78. This case has 6 events. LEGALBERT misses one argument of the first out of two residence_ events, and overpredicts a second start_ date. LEGALBERT mispredicts the patient_ of the second residence_ event, but nonetheless makes a sensible prediction. Together with LEGALBERT', BERT and T5, LEGALBERT correctly detects a payment_ event, even though that event had been discarded as part of pre-processing, as described in Section 5. 1. For the first residence_ event, LEGALBERT' mispredicts Bob as the agent_ instead of Charlie, predicts 2 start_ dates, and 2 patient_ arguments.
For the second residence_ event, LEGALBERT' predicts 2 agent_ and 2 patient_ arguments, one too many in each case. BERT, ROBERTA and T5 make similar predictions, getting all the event anchors right, but missing about half the arguments per event. T A X_C A S E_34. This case contains a single income_ event. All 5 models perfectly predict the gold KB. 8 Precision is higher in Table  4  for dev precision for LEGAL-BERT. This difference comes from one out of the five runs, where AllenNLP returned precisions of 88. 6 and 88. 7. This is unexpected, and can presumably be ascribed to an artefact in rounding in Python. The remaining four runs have identical precision, and this single discrepancy led to different means, 88. 74 and 88. 76. T A X_C A S E_28. This case contains 3 events. LEGALBERT misses a token when predicting the amount_ argument of the payment_ event: it predicts "53" instead of "53200".
De novo AMP design using deep learning For de novo-design of AMPs, we adapted two versions of deep generative variational autoencoders (VAE) from a previous study  23. Generative VAE are unsupervised learning models, which take as input only AMP sequences, and comprise an encoder, a latent space, and a decoder. During model training, the encoder compresses the input sequences into a lowdimensional space ("latent space"), while the decoder aims at reconstructing sequences from this latent space (Fig. 1a ). We first pretrained the VAEs using ~1. 5 million peptide sequences from UniProt as a generic dataset. Second, performed transfer learning on the pretrained VAEs using a dataset of ~5,000 known AMPs to set up the latent space to be used for de novo AMP generation (Methods and Supplementary Table  1 ).
To reduce the number of AMPs for experimental testing, we set up a method to select potential candidates according to their predicted bioactivity, i. e. , their Minimum Inhibitory Concentration (MIC). To that end, we established predictive deep learning models that we trained with the sequence and respective experimental MIC values of ~5,000 known AMPs (sequence-MIC relationship, Fig. 1b ). As regressors, we used convolutional neural networks (CNN) and recurrent neural networks (RNN) (Methods and Supplementary Table  1 ). To identify interesting AMP candidates, we first generated new AMPs by sampling points from the latent space and subsequently feeding them into the decoder, which yielded peptide sequences that share the same properties but are novel compared to the training dataset. These de novo AMPs were then prioritized by the regressors according to their predicted MICs.
5a ). Phenylalanine was particularly overrepresented at the cost of leucine, which was underrepresented (Supplementary Fig. 5b , Supplementary Table  3 ). BLAST searching showed that our de novo AMPs were unique in their sequence. No significant similarity was observed against the UniProt database, encompassing ~240 million entries, nor an AMP sequence from the training data set. (Supplementary Tables  4 5 6 , Supplementary Note 2 for detailed BLAST sequence similarity analyses). Altogether, these results demonstrated that our de novo AMPs shared the physico-chemical building principles with their natural counterparts, but were distinct from them in their amino acid sequences.
Pretraining and training datasets Pretraining data. To gather a large corpus of protein sequences representing a general protein grammar, we downloaded all protein sequences shorter than 49 amino acids from UniProt  34  (as of July 2021). After removing duplicate sequences and entries with unknown amino acid characters, 3,104,952 unique sequences remained of which a random subset of half of them was used for pretraining. AMP data. For experimentally validated AMP sequences, we used the Giant Repository of AMP Activities (GRAMPA)  35  which has combined sequence and activity data from several public AMP databases; APD  36  , DADP  37  , DBAASP  38  , DRAMP  39  , and YADAMP  40. This database consists of 6,760 unique sequences and 51,345 total MIC measurements, spanning several bacterial and nonbacterial target species.
We filtered the MIC measurements to the ten most abundant bacterial species (E. coli, P. aeruginosa, Salmonella typhimurium, K. pneumoniae, A. baumannii, S. aureus, B. subtilis, S. epidermidis, Micrococcus luteus and E. faecalis) and omitted all peptides containing any chemical modification other than C-terminal amidation, for feasible in-laboratory expression. After removing duplicate sequences and entries longer than 48 amino acids, 5,319 unique AMP sequences were left. Non-AMP data. We searched the UniProtKB  34, 41  for proteins labeled as "NOT antimicrobial, antibiotic, antiviral or antifungal" (downloaded as of July 2021), removed entries containing ambiguous amino acids, and kept only unique sequences shorter than 49 amino acids. This resulted in a dataset containing 10,612 unique non-AMP sequences.
Generator variational autoencoder (VAE) We used VAEs because they have previously been used for de novo AMP design  16, 17, 19, 42. The generative VAE consists of an encoder, a latent vector, and a decoder. The encoder feeds the input data (one-hot encoded amino acid letter of peptides) into a latent vector that is an information bottleneck, and the decoder aims to reconstruct the input data from the latent vector. Training the VAE and minimizing the difference between the input data and the reconstructed data acts twofold; the encoder learns to map the training dataset into a lower-dimensional space and the decoder learns to generate samples similar to the training data from any vector in the latent space. Thus, each peptide in the training dataset lands on a point in the multi-dimensional latent space.
Picking vectors from the empty regions in this space and feeding them into the decoder yield peptide sequences that share the same grammar but are novel and not seen in the training dataset. After pretraining and training (transfer learning), we generated new AMPs by sampling from the latent space using different strategies in particular by exploring the neighborhood of a control functional AMP, gradient descent, or random sampling (Supplementary Tables  1, 2 ). Generator VAE models. We adapted the neural network architecture of the CNN-RNN hybrid VAE model from Hawkins-Hooker et al. 23. The encoder consists of 5 consecutive one-dimensional convolution layers fed into a dense layer of size 50, which is the latent vector. The decoder is made from 4 deconvolution layers that samples the latent vector and a GRU layer of 512 cells outputs a sequence in the same dimension as the input.