dataesr/scientific-paragraphs-categorization huggingface dataset:
Benchmark results We evaluate three existing NER models on our dataset: (1) the standard BiLSTM-CRF model which can handle only flat entities  (Lample et al. , 2016) ; (2) hypergraph-based  (Wang and Lu, 2018) ; and, (3) transition-based  (Wang et al. , 2018)  models. The latter two models were proposed to recognize nested mentions. We follow CoNLL evaluation schema in requiring an exact match of mention start, end and entity type  (Sang and Meulder, 2003). We use sections 02 as development set, sections 23 and 24 as test set, and the remaining sections as training set. The model that performs best on the development set is evaluated on the test set for the final result. Since the standard BiLSTM-CRF model cannot handle nested entities, we use either the outermost (BiLSTM-CRF-TOP in Table  2 ) or the innermost mentions (BiLSTM-CRF-BOTTOM) for training.
We also combine the outputs from these two flat NER models, and denote the result as BiLSTM-CRF-BOTH. From Table  2 , we can see that single flat NER models can achieve high precision but suffer from low recall. For example, the model pretrained on outermost (top) mentions has 38. 0 recall, as   Wang and Lu, 2018). However, its decoding speed of 9 words per second is slow due to the large number of entity categories of our dataset. 2 The transition-based method has a higher decode speed of 57 words per second, but has much lower precision than flat NER models.
Related Work Other corpora with nested entities: We briefly compare existing annotated English corpora involving nested entities. A comparison of statistics between our dataset and two widely used benchmark datasets is shown in Table  3. The ACE corpora  (Mitchell et al. , 2004; Walker et al. , 2005)  consist of data of various types annotated for entities, relations and events. disease names, often nest. For example, the RNA 'CIITA mRNA' contains a DNA mention 'CIITA'. In addition to these two commonly used nested entity corpora,  Byrne (2007)  and  Alex et al. (2007)  introduced datasets with nested entities in historical archive and biomedical domains, respectively. However, their datasets are not publicly available. Four percent of entity mentions annotated in the English entity discovery and linking task in TAC-KBP track include nesting  (Ji et al.
, 2014). Resources built on the PTB: A lots of effort has been made on adding syntactic and semantic information to the PTB  (Marcus et al. , 1993). PropBank  (Kingsbury et al. , 2002)  extended the PTB with the predicate argument relationships between verbs and their arguments. NomBank  (Meyers et al. , 2004)  extended the argument structure for instances of common nouns. Vadas and Curran (2007) , and  Ficler and Goldberg (2016)  extended the PTB with noun phrase and coordination annotations, respectively. Our dataset is built on top of the PTB and enriches the full ecosystem of resources and systems that stem from it.
Summary We present NNE, a large-scale, nested, finegrained named entity dataset. We are optimistic that NNE will encourage the development of new NER models that recognize structural information within entities, and therefore understand finegrained semantic information captured. Additionally, our annotations are built on top of the PTB, so that the NNE dataset will allow joint learning models to take advantage of semantic and syntactic annotations, and ultimately to understand and exploit the true structure of named entities.
Simulation data The simulation considered a population consisting of 1000 males and 1000 females over 10 non-overlapping generations (Supplementary File 1). Dams were sampled randomly (with replacement) from the 1000 females, and lists of 100 standard sires and 10 popular sires (sires producing a greater proportion of offspring) were sampled from the 1000 males. Five different scenarios were considered (Table  1 ). In scenario a, male reproducers were randomly chosen within the sire lists, with popular sires producing 70% of the offspring. In scenario b, to simulate a selection process involving nonindependence in family size across generations  (Hill 1979) , selection on a given trait was considered, assuming an additive polygenic model (see Supplementary File 1). Lists of popular sires and standard sires were determined at each generation by selecting the 10 + 100 males with the greatest estimated breeding values (EBVs).
We present the design of a new functional programming language, MLTS, that uses the λ-tree syntax approach to encoding bindings appearing within data structures. In this approach, bindings never become free nor escape their scope: instead, binders in data structures are permitted to move to binders within programs. The design of MLTS includes additional sites within programs that directly support this movement of bindings. In order to formally define the language's operational semantics, we present an abstract syntax for MLTS and a natural semantics for its evaluation. We shall view such natural semantics as a logical theory within a rich logic that includes both nominal abstraction and the ∇-quantifier: as a result, the natural semantics specification of MLTS can be given a succinct and elegant presentation.
variable names or, in a more abstract way, by using de Bruijn indexes  [10]. Since such techniques are quite complex to get right and since bindings are so pervasive, a great deal of energy has gone into making tools and libraries that can help deal with binders: for example, there is the locally nameless approach  [4, 21, 30]  and the parametric higher-order abstract syntax approach  [8]. Extending a functional programming language with features that support bindings in data has been considered before: for example, there have been the FreshML  [53, 59]  and CαML  [52]  extensions to ML-style functional programming languages. Also, entirely new functional programming languages, such as the dependently typed Beluga  [47]  language, have been designed and implemented with the goal of supporting bindings in syntax.
In the domains of logic programming and theorem proving, several designs and implemented systems exist that incorporate approaches to binding: such systems include Isabelle's generic reasoning core  [43] , λProlog  [37, 40] , Qu-Prolog  [6] , Twelf  [46] , αProlog  [5] , the Minlog prover  [57] , and the Abella theorem prover  [3]. In this paper we present MLTS, a new language that extends (the core of) ML and incorporates the λ-tree syntax approach to encoding the abstract syntax of data structures containing binders. Briefly, we can define the λ-tree syntax approach to syntax as following the three tenets:  (1)  Syntax is encoded as simply typed λ-terms in which the primitive types are identified with syntactic categories. (2) Equality of syntax must include α βη-conversion. (3) Bound variables never become free: instead, their binding scope can move.
This latter tenet introduces the most characteristic aspect of λ-tree syntax which is often called binder mobility. MLTS is, in fact, an acronym for mobility and λ-tree syntax. This paper contains the following contributions. -We present the design of MLTS, a new functional language prototype for dealing with bindings, aiming at expressivity and generality. -We show how the treatment of bindings that has been successful in the logic programming and theorem proving systems λProlog, Twelf, and Abella, can be incorporated into a functional programming language. At the same time, MLTS remains a ML-family language; nominals are treated similarly to constructors of algebraic datatypes (in expressions and patterns), distinguishing our design from existing proposals, such as Delphin and Beluga. -We present some of the metatheory of MLTS.
A TOUR OF MLTS We chose the concrete syntax of MLTS to be an extension of that of the OCaml programming language (a program in MLTS not using the new language features should be accepted by the ocamlc compiler). We assume that the reader is familiar with basic syntactic conventions of OCaml  [42] , many of which are shared with most ML-like programming languages. This section presents the new constructs of MLTS along with a set of examples. We also provide a web application, TryMLTS  [20] , that can serve as a companion during the reading of this introduction to the language.
Current prototype implementation We have a prototype implementation of MLTS. A parser from our extended OCaml syntax and a transpiler that generates λProlog code are implemented in OCaml. A simple evaluator and type checker written in λProlog are used to type-check and execute the generated MLTS code. The implementation of the evaluator in λProlog is rather compact but not completely trivial since the natural semantics of MLTS (presented in Section 3. 4) contains features (namely, ∇-quantification and nominal abstraction) that are not native to λProlog: they needed to be implemented. Both the Teyjus  [54]  and the Elpi  [12]  implementations of λProlog can be used to execute the MLTS interpreter. Since Elpi, the parser, and the transpiler are written in OCaml, web-based execution was made possible by compiling the OCaml bytecode to a Javascript client library with js_of_ocaml [27].
Church's integration is also a popular one in theorem proving-being the core logic of the Isabelle  [44] , HOL  [23, 25] , and Abella  [3]  theorem provers-as well as the logic programming language λProlog  [37]. Given the existence of these provers, a good literature now exists that describes how to effectively implement STT and closely related logics. Since the formal specifications of evaluation and typing will be given using inference rules and since such rules can be viewed as quantified formulas, this literature provides means for implementing MLTS.
Natural semantics specification of MLTS We can now define the operational semantics of MLTS by giving inference rules in the style of natural semantics (a. k. a. big-step semantics) following Kahn  [28]. The semantic definition for the core of MLTS is defined in Figure  7. Since those inference rules are written using a higher-order abstract syntax for MLTS, directly inspired by λProlog term representations; we briefly describe how that abstract syntax is derived from the concrete syntax. Instead of detailing the translation from concrete to abstract syntax, we illustrate this translation with an example. There is an implementation of MLTS that includes a parser and a transpiler into λProlog code: this system is available for online use and for download at https://trymlts. github. io [20].
Typing Given that MLTS is a rather mild extension of OCaml at the syntax level, a typing system for MLTS is simple to present and follows standard practices. Figure  9  contains the rules for typing the new features of MLTS: additional rules for encoding let and let rec constructions (as well as for built-in types such as integers) must also be added, but these follow the usual pattern. The inference rules in this figure involve the following typing judgments. Γ ⊢ M : A Γ ⊢ A : R : B Γ ⊢ M : A ⊣ ∆ open A In all of these rules, Γ is the usual association between bound variables and a type: in our situation, Γ will associate both variables and nominals to type expressions. (We also assume that the order of pairs in Γ is not important. ) The first of these judgments is the usual typing judgment between a program expression M and A.
Small-step operational semantics As a complement to the natural (big-step) semantics of Figure  7 , we developed a small-step operational semantics of MLTS. Its two salient features are as follows: (1) the small-step treatment of evaluation contexts clarifies the moments during reduction where escapechecking must be performed (this is often left implicit in the natural semantics), and (2) its treatment of pattern-matching does not use nominal-abstraction -it implements an equivalent but lowerlevel mechanism. This lower-level expression of the handling of nabla-bound nominals in pattern-matching gives a more operational intuition of the language, and it also guides practical implementations in languages without native support for nominal abstraction. In fact, we co-evolved this operational semantics with the λProlog implementation of the language, the former guiding the latter, with the bugs found playing with the latter informing changes to the former -using the natural semantics as a reference specification for what the behavior should be.
Formal properties of MLTS Given the restrictions of Section 4. 2, we can list the following three formal properties about MLTS. ). Let E be the abstract syntax of an MLTS program that does not contain any free nominal. If ⊢ E ⇓ V is provable then V does not contain any free nominals. The proof of this follows from a simple induction on the structure of proofs in the logic G: the precise nature of the semantic specification given in Figure  7  is not relevant. The systematic use of the ∇-quantifier guarantees this conclusion.
BINDER MOBILITY We started this programming language project with the desire to treat binders in syntax as directly and naturally as possible. We approached this project by designing the MLTS language with more binders than, say, OCaml: it has not only the usual binders for building functions and for refactoring computation (via the let construction) but also new binders that are directly linked to binders in data (via the new X in, nab X in, and X\ operators). Finally, the natural semantics of MLTS in G and its implementation in λProlog are all based on using logics that contain rich binding operators that go beyond the usual universal and existential quantifiers. It is worth noting that if one were to write MLTS programs that do not need to manipulate data structures containing bindings, then the new binding features of MLTS would not be needed and neither would the novel features of both G and λProlog.
Thus, in a sense, binders have not been formally implemented in this story: instead, binders of one kind have been implemented and specified using binders in another system. We were able to complete a prototype implementation of MLTS since the implementers of λProlog provide a low-level implementation of bindings that we are able to use in our static and dynamic semantics specifications. One way to view the processing of a binder is that one first opens the abstraction, processes the result (by "freshening" the freed names), and then closes the abstraction  [52]. In the setting of MLTS, it is better to view such processing as the movement of a binder: that is, the binder in a data structure actually gets re-identified with an actual binder in the programming language. As we illustrated in Section 2.
RELATED WORK The term higher-order abstract syntax (HOAS) was introduced in  [45]  to describe an encoding technique available in λProlog. A subsequent paper identified HOAS as a technique "whereby variables of an object language are mapped to variables in the metalanguage"  [46]. When applied to functional programming, this description implies the mapping of bindings in syntax to the bindings that create functions. Unfortunately, such encoding technique often lacks adequacy (since "exotic terms" can appear  [11] ), and structural recursion can slip away  [15]. The terms λ-tree syntax  [36, 38]  and binder mobility  [35]  were later introduced to describe the different and more syntactic approach that we have used here.
Systems with two arrow type constructors The ML λ  [31]  extension to ML is similar to MLTS in that it also contains two different arrow type constructors (-> and =>) and pattern matching was extended to allow for pattern variables to be applied to a list of distinct bound variables. The new operator of MLTS could be emulated by using the backslash operator and the "discharge" function of ML λ. Critically missing from that language was anything similar to the nab binding of MLTS. Also, no formal specification and no implementation were ever offered. Licata & Harper  [29]  have used the universe feature of Agda 2 to provide an implementation of bindings in data structures that also relies on supporting two different implications-as-types. Nominals and nominal abstraction, in the sense used in this paper, were first conceived, studied, and implemented as part of the Abella theorem prover  [3].
Systems with one arrow type constructor The Delphin design is probably the closest to MLTS, in particular  [56]  introduced a programming-language version of the ∇ quantifier from  [39] , whose usage is related to the ∇ of MLTS. In Delphin, ∇ introduces normal term variables (there is no separate class of nominal constants), while MLTS presents nominals as closer to datatype constructors, with a natural usage in pattern-matching. Delphin makes nominal-escape errors impossible at runtime by imposing a static discipline to prevent them, while MLTS allows runtime failure in order to allow for more experimentation. The original proposal in  [56]  uses a type modality that imposes a strict FIFO discipline on free variables. This discipline was found too constraining;  [50]  completely eschews a new construct (its νx.
Static checks of Beluga programs can be used to prove the formal correctness of Beluga programs (commonly by proving that a given piece of program code is, in fact, a total function). As a result, a checked Beluga program is often a formal proof. Since a wide range of formal systems can be encoded naturally using dependently typed λ-terms  [1, 24] , Beluga programs can be used for both programming with and reasoning about the meta-theory of those formal systems. Since bindings and contexts are part of the vocabulary of Beluga, these formal proofs can capture the metatheory of logical and computational systems (such as natural deduction proof systems and the operational semantics of rich programming languages). The goal of MLTS is intended only to support programming and not directly reasoning: the intent of the new features of MLTS is only to support the manipulation of syntax containing bindings.
Systems using nominal logic The FreshML  [59]  and CαML  [52]  functional programming languages provide an approach to names based on nominal logic  [48]. These two programming languages provide for an abstract treatment of names and naming. Once naming is available, binding structures can also be implemented. In a sense, the design of these two ML-variants are also more ambitious than the design goal intended for MLTS: in the latter, we were not focused on naming but just bindings. The recent paper  [14]  introduces a syntactic framework that treats bindings as primitives. That framework is then integrated with various tools and with the framework of contextual types (similar to that found in Beluga) in order to provide a programmer of, say, OCaml with sophisticated tools for the manipulation of syntax and binders.
CONCLUSION While the λ-tree syntax approach to computing with syntax containing bindings has been successfully developed within the logic programming setting (in particular, in λProlog and Twelf), we provide in this paper another example of how binding can be captured in a functional programming language. Most of the expressiveness of MLTS arises from its increased use of program-level binding. The sophistication needed to correctly exploit binders and quantifiers in MLTS is a skill most people have learned from using quantification in, for example, predicate logic. We have presented a number of MLTS programs and we note that they are both natural and unencumbered by concerns about managing bound variable names. We have also presented a typing discipline for MLTS as well as a formal specification of its natural semantics: this latter task was aided by being able to directly exploit a rich logic, called G, that allows capturing both λ-tree syntax and binder mobility.
In addition to xenobiotic substrates, CES1 is also involved in the processing of several endogenous compounds such as fatty acids and cholesterol derivatives  [15, 16]. The publication of crystal structures of CES1 has greatly contributed to the understanding of the mechanism of action of the enzyme. Fourteen X-ray structures of CES1 released in the Research Collaborator for Structural Bioinformatics (RCSB) Protein Databank were all complexed with a ligand (substrate analogs, inhibitors, endogenous metabolites, and enzymatic products)  [17] [18] [19] [20]. In addition, four new X-ray structures in the apo state have been resolved recently  [21, 22]. Tremendous efforts have been devoted by examining both structural and biochemical requirements of these enzymes to hydrolyze their substrates  [23] [24] [25] , and several early studies reported different types of CESs inhibitors  [26] [27] [28] [29] [30] [31] [32] [33].
This composition confers the ability to act on structurally diverse compounds. Figure  1  presents the human CES1 trimer, the active site composition and the binding modes of CES1 with the co-crystallized ligand naloxone (PDB ID 1MX9). The catalytic triad residues, located between the two pockets, are aligned in a way that favors the generation of the Ser221 oxygen nucleophile. This nucleophile then attacks the carbonyl carbon of the ester substrate, leading to the formation of the acyl-enzyme intermediate, which is stabilized by the oxyanion hole residues Gly142 and Gly143. Additionally, the enzyme has two other ligand-binding sites referred to as the side-door and the Z-site: The side-door serves as an alternative/additional opening to the active site for substrate entrance or as a product release pore: The Z-site probably has an allosteric function by modulation of the binding site accessibility  [45, 46].
This composition confers the ability to act on structurally diverse compounds. Figure  1  presents the human CES1 trimer, the active site composition and the binding modes of CES1 with the cocrystallized ligand naloxone (PDB ID 1MX9). The catalytic triad residues, located between the two pockets, are aligned in a way that favors the generation of the Ser221 oxygen nucleophile. This nucleophile then attacks the carbonyl carbon of the ester substrate, leading to the formation of the acyl-enzyme intermediate, which is stabilized by the oxyanion hole residues Gly142 and Gly143. Additionally, the enzyme has two other ligandbinding sites referred to as the side-door and the Z-site: The side-door serves as an alternative/additional opening to the active site for substrate entrance or as a product release pore: The Z-site probably has an allosteric function by modulation of the binding site accessibility  [45, 46].
Analysis of the Docking Protocol with Known CES1 Ligands Once the docking tool was validated, exhaustive docking for a set of 294 compounds with a known inhibition constant (Ki) at CES1 was completed with the 10 X-ray structures. To analyze the performance of the docking tool, the ROC curve (Receiver Operating Characteristic (ROC) was used, considering the Vina docking score and a 10 µM Ki threshold, splitting the dataset into an active and inactive group, with 140 and 154 ligands respectively. An ensemble docking approach was also considered, by taking the average of the scores of the 10 protein structures (Figure  2 ). The ROC curves were similar for seven conformations of the enzyme, showing a general discrimination ability of relatively weak size (AUC around 0. 57-0. 62). Two conformations remained close to the center line, with AUC 0.
51 and 0. 52, meaning that inactive compounds were docked in CES1 with a Vina docking score as good as for the active compounds. One structure (1YAH) showed inverse discriminative power (AUC 0. 38), corresponding to the worst docking scores for the more active compounds. The ensemble docking (red line in Figure  2 ) does not show a better performance (AUC of 0. 57) in comparison to many individual structures. Simple averaging of the score across all structure does not yield better performance. Seeking to explain such behavior, we noticed that 1YAH binds to the smallest substrate by far (ethyl acetate), with a corresponding small active site volume and exposed surface area. Small molecules like ethyl acetate were not found among the known high potency CES1 ligands in our dataset, thus high-scoring compounds on this protein structure were unlikely to be legitimate hits.