dataesr/scientific-paragraphs-categorization huggingface dataset:
Acknowledgements The authors would like to thank  Laurence Van Melderen  and  Ariane Toussaint  for providing strain  RH8816 ,  Erica Lieberman  from  Eligo Bioscience  for providing the sequencing data for phage HK97 and  Laurent Debarbieux  for providing phage T4 as well as useful discussions. We also acknowledge the  Genomics Platform and the center of informatics for Biology , especially  Christiane Bouchier ,  Laurence Ma ,  Eric Deveau ,  Fabien Mareuil  and  Olivia Doppelt-Azeroual  for their help and support with sequencing and Galaxy integration. This work was supported by a discovery grant from the  Natural Sciences and Engineering Research Council of Canada  (NSERC # 341450-2010 ) to LCF and the  French Government's program Investissement d' Avenir program ;  Laboratoire d'Excellence 'Integrative Biology of Emerging Infectious Diseases ' [ ANR-10-LABX-62-IBEID].
We propose a core semantics for Dependent Haskell, an extension of Haskell with full-spectrum dependent types. Our semantics consists of two related languages. The first is a Curry-style dependently-typed language with nontermination, irrelevant arguments, and equality abstraction. The second, inspired by the Glasgow Haskell Compiler's core language FC, is its explicitly-typed analogue, suitable for implementation in GHC. All of our resultsÐchiefly, type safety, along with theorems that relate these two languagesÐhave been formalized using the Coq proof assistant. Because our work is backwards compatible with Haskell, our type safety proof holds in the presence of nonterminating computation. However, unlike other full-spectrum dependently-typed languages, such as Coq, Agda or Idris, because of this nontermination, Haskell's term language does not correspond to a consistent logic.
The zip function type checks because in the first branch n is equal to zero, so Nil has a type equal to Vec a n. In the second branch, when n is equal to S m, then the result of the recursive call has type Vec a m, so the result type of the branch is Vec a (S m), also equal to Vec a n. This pattern matching is exhaustive because the two vectors have the same length; the two remaining patterns are not consistent with the annotated type. In an explicitly-typed core language, such as DC, we use typing annotations to justify this reasoning. First, consider the elaborated version of the Vec datatype definition shown below. This definition explicitly binds the argument m to the (:>) constructor, using forall to note that the argument need not be stored at runtime. Furthermore, the type of each data constructor includes a context of equality constraints, describing the information gained during pattern matching.
The core language version of zip, shown below, uses the binder \-to abstract irrelevant arguments and the binder /\ to abstract coercions. The core language does not include nested pattern matching, so the case analysis of each list must be done separately. Each case expression includes two branches, one for each data constructor (Nil and (:>)). Each branch then quantifies over the arguments to the matched data constructor, including coercions. For example, (:>) above takes four arguments, the implicit length m, the coercion (n ~S m), the head of the vector (of type a) and the tail of the vector (of type Vec a m). (:>) [a][n][m1][c1] ((,) [a][b] x y) (zip [m1][a][b] xs (ys |> Vec b (nth 2 [sym c2; c1])) The core language zip function must provide all arguments to data constructors and functions, even those that are inferred in the source language.
MECHANIZED METATHEORY All of the definitions, lemmas and proofs in this paper are mechanized in the Coq proof assistant [Coq development team 2004], using tactics from the ssreflect library  [Gonthier et al. 2016 ]. We used the Ott tool  [Sewell et al. 2010]  to generate both the typeset rules in this paper and the Coq definitions that were the basis of our proofs. Our formalization uses a locally nameless representation of terms and variable binding. Some of the proofs regarding substitution and free variables were automatically generated from our language definition via the LNgen tool  [Aydemir and Weirich 2010]. Our total development includes our 1,400 line Ott specification, about 17,000 nonblank lines of Coq proofs, plus another 13,000 lines of Coq generated by Ott and LNgen.
ACKNOWLEDGMENTS Thanks to  Simon Peyton Jones ,  Adam Gundry ,  Iavor Diatchki  and  Pritam Choudhury  for feedback and suggestions. This material is based upon work supported by the  National Science Foundation  under Grant No. 1319880  and Grant No. 1521539. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the  National Science Foundation.
VMWEs of different categories may share the same syntactic structure and lexical choices (to make a mistake is a light-verb construction, to make a meal is an idiom), 5. VMWEs behave differently in different languages and are modelled according to different linguistic traditions. These properties are challenging for automatic identification of VMWEs, which is a prerequisite for MWE-aware downstream applications such as parsing and machine translation. Namely, challenge 1 hinders the use of traditional sequence labelling approaches and calls for syntactic analysis. Challenges 2, 3 and 4 mean that VMWE identification and categorization cannot be based on solely syntactic patterns. Challenge 5 defies crosslanguage VMWE identification. We present an initiative aiming at boosting VMWE identification in a highly multilingual context.
It is based on a joint effort, carried on within a European research network, to elaborate universal terminologies, guidelines and methodologies for 18 languages. Its main outcome is a 5million-word corpus annotated for VMWEs in all these languages, which underlies a shared task on automatic identification of VMWEs. 2 Participants of the shared task were provided with training and test corpora, and could present systems within two tracks, depending on the use of external resources. They were encouraged to submit results for possibly many covered languages. In this paper, we describe the state of the art in VMWE annotation and identification ( § 2). We then present the corpus annotation methodology ( § 3) and its outcome ( § 4). The shared task organization ( § 5), the measures used for system evaluation ( § 6) and the results obtained by the participating systems ( § 7) follow.
Related Work Annotation There have been several previous attempts to annotate VMWEs. Some focus specifically on VMWEs and others include them among the linguistic phenomena to be annotated. Rosén et al. (2015)  offer a survey of VMWE annotation in 17 treebanks, pointing out that, out of 13 languages in which phrasal verbs do occur, 8 have treebanks containing annotated phrasal verbs, and only 6 of them contain annotated light-verb constructions and/or verbal idioms. They also underline the heterogeneity of these MWE annotations. Nivre and  Vincze (2015)  show that this is also the case in the treebanks of Universal Dependencies (UD), despite the homogenizing objective of the UD project  (McDonald et al. , 2013). More recent efforts  (Adalı et al. , 2016) , while addressing VMWEs in a comprehensive way, still suffer from missing annotation standards.
, 2016)  and English noun compounds  (Reddy et al. , 2011). In contrast to these seminal efforts, the present shared task relies on VMWE annotation in running text according to a unified methodology. Identification MWE identification is a wellknown NLP task. The 2008 MWE workshop proposed a first attempt of an MWE-targeted shared task. Differently from the shared task described here, the goal of participants was to rank provided MWE candidate lexical units, rather than to identify them in context. True MWEs should be ranked towards the top of the list, whereas regular word combinations should be at the end. Heterogeneous datasets containing several MWE categories in English, German and Czech were made available. Two systems participated, using different combinations of features and machine learning classifiers.
In addition to the shared task, the MWE 2008 workshop also focused on gathering and sharing lexical resources containing annotated candidate MWEs. This repository is available and maintained on the community website. 3 he DiMSUM 2016 shared task  (Schneider et al. , 2016)  challenged participants to label English sentences (tweets, service reviews, and TED talk transcriptions) both with MWEs and supersenses for nouns and verbs. 4 The provided dataset is made of approximately 90,000 tokens containing 5,069 annotated MWEs, about 10% of which are discontinuous. They were annotated following  Schneider et al. (2014b) , and thus contain several VMWEs types on top of non-verbal MWEs. Links between MWE identification and syntactic parsing have also long been an issue. While the former has often been treated as a pre-processing step before the latter, both tasks are now more and more often integrated, in particular for continuous MWE categories  (Finkel and Manning, 2009; Green et al.
Annotation Guidelines The biggest challenge in the initial phase of the project was the development of the annotation guidelines 7  which would be as universal as possible but which would still allow for languagespecific categories and tests. To this end, a twophase pilot annotation in most of the participating languages was carried out. Some corpora were annotated at this stage not only by native but also by near-native speakers, so as to promote cross-language convergences. Each pilot annotation phase provided feedback from annotators and was followed by enhancements of the guidelines, corpus format and processing tools. In this way, the initial guidelines dramatically evolved, new VMWE categories emerged, and the following 3level typology was defined: 1. universal categories, that is, valid for all languages participating in the task: (a) light verb constructions (LVCs), e.
Consistency Checks and Homogenisation Even though the guidelines heavily evolved during the two-stage pilot annotation, there were still questions from annotators at the beginning of the final annotation phase. We used an issue tracker system (gitlab) in which language leaders could share questions with other language teams. High-quality annotation standards require independent double annotation of a corpus followed by adjudication, which we could not systematically apply due to time and resource constraints. For most languages each text was handled by one annotator only (except for a small corpus subset used to compute inter-annotator agreement, cf. § 4. 2). This practice is known to yield inattention errors and inconsistencies between annotators, and since the number of annotators per language varies from 1 to 10, we used consistency support tools.
Firstly, some languages (BG, FR, HU, IT, PL, and PT) kept a list of VMWEs and their classification, agreed on by the annotators and updated over time. Secondly, some languages (DE, ES, FR, HE, IT, PL, PT, and RO) performed a step of homogenisation once the annotation was complete. An in-house script read the annotated corpus and generated an HTML page where all positive and negative examples of a given VMWE were grouped. Entries were sorted so that similar VMWEs appear nearby -for instance occurrences of pay a visit would appear next to occurrences of receive a visit. In this way, noise and silence errors could easily be spotted and manually corrected. The tool was mostly used by language leaders and/or highly committed annotators.
Corpora Tables  4  and 5  provide overall statistics of the training and test corpora created for the shared task. We show the number of sentences and tokens in each language, the overall number of annotated VMWEs and the detailed counts per category. In total, the corpora contain 230,062 sentences for training and 44,314 sentences for testing. These correspond to 4,5M and 900K tokens, with 52,724 and 9,494 annotated VMWEs. The amount and distribution of VMWEs over categories varies considerably among languages. No category was used in all languages but the two universal categories, ID and LVC, were used in almost all languages. In HU, no ID was annotated due to the genre of the corpus, mainly com-posed of legal texts. In FA, no categorisation of the annotated VMWEs was performed, therefore, the OTH category has special semantics there: it does not mean that a VMWE cannot be categorised because of its linguistic characteristics, but rather that the categorisation tests were not applied.
g. auf|machen (lit. out|make) 'open'. The right part of Table  6  shows the length of discontinuities. The data sets vary greatly across languages. While for BG, FA and IT, more than 80% of VMWEs are continuous, for DE, 30. 5% of VMWEs have discontinuities of 4 or more tokens. All the corpora are freely available. The VMWE annotations are released under Creative Commons licenses, with constraints on commercial use and sharing for some languages. Some languages use data from other corpora, including additional annotations ( § 5). These are released under the terms of the original corpora.
Format The official format of the annotated data is the parseme-tsv format 10 , exemplified in Figure  1. It is adapted from the CoNLL format, with one token per line and an empty line indicating the end of a sentence. Each token is represented by 4 tabseparated columns featuring (i) the position of the token in the sentence or a range of positions (e. g. , 1-2) in case of multiword tokens such as contractions, (ii) the token surface form, (iii) an optional 10 http://typo. uni-konstanz. de/parseme/index. php/2-general/ 184-parseme-shared-task-format-of-the-final-annotation nsp flag indicating that the current token is adjacent to the next one, and (iv) an optional VMWE code composed of the VMWE's consecutive number in the sentence and -for the initial token in a VMWE -its category (e. g. , 2:ID if a token starts an idiom which is the second VMWE in the current sentence).
In case of nested, coordinated or overlapping VMWEs multiple codes are separated with a semicolon. Formatting of the final corpus required a language-specific tokenisation procedure, which can be particularly tedious in languages presenting contractions. For instance, in FR, du is a contraction of the preposition de and the article le. Some language teams resorted to previously annotated corpora which have been converted to the parseme-tsv format automatically (or semiautomatically if some tokenisation rules were revisited). Finally, scripts for converting the parseme-tsv format into the FoLiA format and back were developed to ensure corpus compatibility with FLAT. Note that tokenisation is closely related to MWE identification, and it has been shown that performing both tasks jointly may enhance the quality of their results  (Nasr et al.
The cases of HE, and especially of IT, should be studied more thoroughly in the future. Note also that in some languages the numbers from a lower bound for the quality of the final corpus, due to post-annotation homogenisation ( § 3. 3). A novel proposal of the holistic γ measure (Mathet et al. , 2015) combines unitising and categorization agreement in one IAA score, because both annotation subtasks are interdependent. In our case, however, separate IAA measures seem preferable both due to the nature of VMWEs and to our annotation methodology. Firstly, VMWEs are know for their variable degree of non-compositionality, i. e. their idiomaticity is a matter of scale. Current corpus annotation standards and identification tools require the MWEhood, conversely, to be a binary property, which sub-optimally models a large number of grey-zone VMWE candidates.
Shared Task Organization Corpora were annotated for VMWEs by different language teams. Before concluding the annotation of the full corpora, we requested language teams to provide a small annotated sample of 200 sentences. These were released as a trial corpus meant to help participants develop or adapt their systems to the shared task particularities. The full corpora were split by the organizers into train and test sets. Given the heterogeneous nature and size of the corpora, the splitting method was chosen individually for each language. As a general rule, we tried to create test sets that (a) contained around 500 annotated VMWEs and (b) did not overlap with the released trial data. When the annotated corpus was small (e. g. in SV), we favoured the size of the test data rather than of the training data, so as to lessen the evaluation bias.
For all languages except BG, HE and LT, we also released companion files in a format close to CONLL-U 14. They contain extra linguistic information which could be used by systems as features. For CS, FA, MT, RO and SL, the companion files contain morphological data only (lemmas, POS-tags and morphological features). For the other languages, they also include syntactic dependencies. Depending on the language, these files were obtained from existing manually annotated corpora and/or treebanks such as UD, or from the output of automatic analysis tools such as UD-Pipe 15. A brief description of the companion files is provided in the README of each language. The test corpus was turned into a blind test corpus by removing all VMWE annotations. After its release, participants had 1 week to provide the predictions output by their systems in the parseme-tsv format.
Predicting VMWE categories was not required and evaluation measures did not take them into account ( § 6). Participants did not need to submit results for all languages and it was possible to predict only certain MWE categories. Each participant could submit results in the two tracks of the shared task: closed and open. The closed track aims at evaluating systems more strictly, independently of the resources they have access too. Systems in this track, therefore, learn their VMWE identification models using only the VMWE-annotated training corpora and the companion files, when available. Cross-lingual systems which predict VMWE annotations for one language using files provided for other languages were still considered in the closed track. Systems using other knowledge sources such as raw monolingual corpora, lexicons, grammars or language models trained on external resources were considered in the open track.
Given that the density of VMWEs per sentence can vary greatly, and in many languages the majority of sentences do not contain any VMWE, we believe that the macro measures are more appropriate. Note also that the measures in (  2 ) are comparable to the CEAF-M measures  (Luo, 2005)  used in the coreference resolution task. 20 There, mentions are grouped into entities (clusters) and the best bijection between gold and system entities is searched for. The main difference with our approach resides in the fact that, while coreference is an equivalence relation, i. e. each mention belongs to exactly one entity, VMWEs can exhibit overlapping and nesting. This specificity (as in other related tasks, e. g. named entity recognition) necessarily leads to counter-intuitive results if recall or precision are considered alone.
System Results Seven systems participated in the challenge and submitted a total of 71 runs. One system (LATL) participated in the open track and six in the closed track. Two points of satisfaction are that (i) each one of the 18 languages was covered and (ii) 5 of the 7 systems were multilingual. Systems were ranked based on their per-token and per-VMWE F-scores, within the open and the closed track. Results and rankings are reported, by language groups, in Tables  7 8 9 10. Most systems used techniques originally developed for parsing: LATL employed Fips, a rulebased multilingual parser; the TRANSITION system is a simplified version of a transition-based dependency parsing system; LIF employed a probabilistic transition-based dependency parser and the SZEGED system made use of the POS and dependency modules of the Bohnet parser.
The ADAPT and RACAI systems employed sequence labelling with CRFs. Finally, MUMULS exploited neural networks by using the open source library Tensor-Flow. In general, scores for precision are much higher than for recall. This can be explained by the fact that most MWEs occur only once or twice in the corpora, which implies that many of the MWEs of the test data were not observed in the training data. As expected, for most systems their per-VMWE scores are (sometimes substantially) lower than their per-token scores. In some cases, however, the opposite happens, which might be due to frequent errors in long VMWEs. The most popular language of the shared task was FR, as all systems submitted predictions for French MWEs. Based on the numerical results, FA, RO, CS and PL were the easiest languages, i.
e. ones for which the best F-scores were obtained. In contrast, somewhat more modest performance resulted for SV, HE, LT and MT, which is clearly a consequence of the lesser amount of training examples for these languages (see Tab. 4). The results for BG, HE, and LT would probably be higher if companion CONLL-U files with morphological/syntactic data could be provided. This would notably allow systems to neutralize inflection, which is particularly rich in verbs in all of these languages, as well as in nouns and adjectives in the first three of them. FA is an outstanding case (with F-score of the best system exceeding 0. 9) and its results are probably correlated with two factors. Firstly, light verbs are explicitly marked as such in the morphological companion files. Secondly, the density of VMWEs is exceptionally high.
If we assume, roughly, one verb per sentence, almost each FA verb is the head of a VMWE, and the system prediction boils down to identifying its lexicalized arguments. Further analysis of this phenomenon should notably include data on the most frequent POS-tags and functions of the lexicalized verbal arguments (e. g. how often is it a nominal direct object) and the average length of VMWEs in this language. Another interesting case is CS, where the size of the annotated data is considerable. This dataset was obtained by adapting annotations from the Prague Dependency Treebank (PDT) to the annotation guidelines and formats of this shared task  (Uresová et al. , 2016; Bejček et al. , 2017). PDT is a long-standing treebank annotation project with advanced modelling and processing facilities. From our perspective it is as a good representative of a high-quality large-scale MWE modelling effort.
In a sense, the results obtained for this language can be considered a benchmark for VMWE identification tools. The relatively high results for RO, CS and PL might relate to the high ratio of IReflVs in these languages. Since the reflexive marker is most often realised by the same form, (CS) se, (PL) się and (RO) se 'self', the task complexity is reduced to identifying its head verb (often adjacent) and establishing the compositionality status of the bigram. Similar effects would be expected, but are not observed, in SL and BG, maybe due to the smaller sizes of the datasets, and to the missing companion file for BG. Note also the high precision of the leading systems in RO, PL, PT, FR and HU, which might be related to the high proportion of LVCs in these languages, and with the fact that some very frequent light verbs, such as (RO) da 'give', (PL) prowadzić 'carry on', (PT) fazer 'make', (FR) effectuer 'perform' and (HU) hoz 'bring', connect with a large number of nominal arguments.