dataesr/scientific-paragraphs-categorization huggingface dataset:
Given our semantic-driven approach on discourse, we chose not to reuse the LADTB relations set. Instead, we started from the set of relations already defined within past SDRT-like annotation campaigns and we refined them via a specialization/generalization process using both Arabic rhetoric literature and corpus analysis. This is motivated by general considerations for capturing additional relations and by language-specific considerations for adapting previous relations to take into account Arabic specificities. We relied on the previous set of 19 relations defined within the Annodis project  (Afantenos et al. , 2012). In this project, relations were grouped into 7 top-level categories: Causation, Structural, Logic, Reported speech, Exposition/Narration, Elaboration and Commentary. Among these relations, we focused our study on semantic relations that involve entities from the propositional content of the clauses (meta-talk (or pragmatic) relations were discarded).
Five situations may occur: (1) There is an exact correspondence between the semantic of R and its equivalent in the Arabic_set. Then, the relation R is selected and the experts analyzed how R is signalled in the corpus in order to give a preliminary list of its discourse markers. (2) There is only a partial correspondence between the semantic of R and its equivalent in the Arabic_set. Then the relation R is selected and the experts further specified its semantic according to the particularities of the Arabic language. (3) The semantic of R covers different senses in the Ara-bic_set and each sense has its own realization in the corpus. Then, R needs to be specialized. New relations are added and the experts were asked to define their semantics along with their corresponding discourse markers.
Finally, in case of multiple relations (i. e. a couple (a,b) linked by different relations), we built as many instances as the number of relations. We designed thirteen groups of features. The first five contain 5 groups (connectives, arguments, al-masdar, tense and negation, length and distance) following (Al-Saif and Markert, 2011). 10 However, compared to (Al-Saif and Markert, 2011), our features are obtained automatically and are not based on the manual annotations of ATB. The 8 remaining features are composed of punctuation, contextual, lexical and lexico-semantic features that have been used in prior work and whose efficiency for detecting both explicit and implicit relations has been empirically determined. They are however new for the Arabic language. Punctuation features were inspired by  (Huang and Chen, 2011)  and  (DuVerle and Prendinger, 2009).
We have 7 string features. We encode the surface strings and the POS of the first three words for each argument (that is a total of 6 features) as well as the syntactic category of the argument parent. If the argument is represented by a non-complete tree (as given by the Stanford outputs), we extract the category of the parent shared by the first and the last word in the argument. (F3) Al-masdar. This is a binary feature that indicates whether the first or the second word of each argument contains al-masdar construction. Al-masdar is a verbal noun construction, frequent in Arabic that names the action denoted by its corresponding verbs. It is a noun category that expresses events without tense. This construction generally signals discourse relations. For example, al-masdar /bHvA/looking in example (11) explains why Ahmed went to the library.
[Ahmed went to the library] 1 [to look for the mathematics book. ] 2 /sbb/Explanation (1,2) Al-masdar is built from the morphological analyzer Al-Khalil  (Boudlal et al. , 2011)  using well-defined morphological patterns composed of 3 or 4 letter-roots. The patterns can attach suffixes to the root and insert consonant/vowel letters or diacritics into the root. More than 60 morphological patterns can be used to generate al-masdar nouns. (F4) Tense and negation. We use a string feature to encode the tense assigned to each argument (perfect, imperfect, future or none) and a binary feature to test the presence of negation words in each argument. To detect negation, we rely on a manually built lexicon of 10 Arabic negation words, such as /lA/ no and /lm/not. Tense features can help identifying relations from the / zmny/Temporal class, such as the relations /tzAmn/Synchronization, and /trtyb bbT'/Slow ordering.
We think that this can be explained by the low frequency of this relation in the dataset (0. 38%). Another interesting finding is that semantic relation features (F11) boost the results of /mEyp/Parallel by more than 0. 060 over (B3)+(F6) to (F10). Indeed, this relation indicates that two units share the same event and have semantically similar constituents, which is captured by some semantic relations of Arabic WordNet such as Near_syonym. [yDHk>xy] [w AlmqAbl tbky >xty. ] 2 [My brother laughs] 1 [however my sister cries. ] 2 [yDHk>xy] 1 [wybky. ] 2 [My brother laughs] 1  [and cries. ]  2 Finally, Fig. 10  shows that our model fails to predict infrequent relations, like /AstntAj/logical-consequence. /grD/Goal and led to the best Fscores with respectively 0. 851 and 0. 735. When adding embedding features (F8), the F-score of the relation /sbb/Explanation degrades by 0.
Detecting single nucleotide polymorphisms (SNPs) between genomes is becoming a routine task with next-generation sequencing. Generally, SNP detection methods use a reference genome. As non-model organisms are increasingly investigated, the need for reference-free methods has been amplified. Most of the existing reference-free methods have fundamental limitations: they can only call SNPs between exactly two datasets, and/or they require a prohibitive amount of computational resources. The method we propose, DISCOSNP, detects both heterozygous and homozygous isolated SNPs from any number of read datasets, without a reference genome, and with very low memory and time footprints (billions of reads can be analyzed with a standard desktop computer). To facilitate downstream genotyping analyses, DISCOSNP ranks predictions and outputs quality and coverage per allele.
We refer to such methods as hybrid, as they use both de novo assembly and mapping techniques to call SNPs. A major limitation of these methods is that they tend to cumulate problems raised in assembly and problems raised in mapping. An interesting alternative is to directly focus on the assembly of SNPs, without trying to assemble a full reference. We refer to such methods as de novo. Several methods that fall into the de novo category have been developed recently  (8) (9) (10) (11) (12). These methods are based on a de Bruijn graph, i. e. a directed graph where the set of vertices corresponds to the set of words of length k (kmers) contained in the reads, and there is an edge between two k-mers if they overlap on k -1 nucleotides. KISSNP  (8)  is a software that takes as input two sets of reads, and detects SNPs using k-mers that are differentially abundant between the two datasets.
CORTEX VAR  (9)  (here after called CORTEX) detects and genotypes SNPs (as well as indels and larger events), between n datasets. BUBBLEPARSE  (10) , which is based on the same graph structure as CORTEX, detects SNPs that it furthermore classifies into homozygous and heterozygous groups. NIKS  (11)  finds homozygous SNPs between two datasets by performing local de novo assembly around sample-specific k-mers. KSNP V2  (13)  proposes yet another approach that is rather dedicated to SNP phylogeny on microbial species. Even if the SNP detection is performed de-novo, this method is strongly based on a reference sequence (on which putative selected k-mers witnessing a SNP are mapped). These methods all entail unpractical computational costs (in particular in terms of memory) for large and complex genomes.
Even with substantial computational resources, most of them still cannot detect variants in mammaliansize datasets. Therefore, there is still a strong need for robust tools that can detect isolated SNPs without a reference genome. For these reasons, we introduce a new de novo method called DISCOSNP. This method is designed to call isolated SNPs directly from sequenced reads, without a reference genome. As shown in this paper, DISCOSNP opens the way to the discovery of SNPs from large-scale studies, even on a standard desktop computer. DISCOSNP finds and ranks high quality isolated heterozygous or homozygous SNPs from any number of read sets, from 1 to n. It introduces new features to distinguish SNPs from sequencing errors or false positives due to approximate repeats. DISCOSNP can be used for finding high-quality isolated SNPs, either heterozygous, e.
g. to build databases of high-quality markers within and across populations, or homozygous between individuals/strains, e. g. to create discriminant markers. The genotyping can be performed as a downstream analysis, based on the read coverage information it provides. Results presented in this paper show that DISCOSNP outperforms other reference-free SNP detection methods in terms of resources (faster and using at least two orders of magnitude less memory), type and number of input dataset(s) (ability to find both homozygous and heterozygous SNPs from more than two datasets), and quality of the ranking of predicted isolated SNPs. On two whole-genome mouse datasets, DISCOSNP detected 2 million isolated SNPs, of which 84% were also found by a manually tuned pipeline in a previous study. On a S.
Algorithms As depicted in Figure  1  DISCOSNP is composed of two modules, KISSNP2 and KISSREADS. KISSNP2 detects putative SNPs from one or more sets of reads. KISSREADS evaluates the coverage and base quality of the SNPs per read set and ranks them accordingly. These two modules are independent, however, a script automatically pipelines them, so their calls are transparent to the user. KISSNP2 module. Similarly to CORTEX and KISSNP, the KISSNP2 module is based on a de Bruijn graph. A de Bruijn graph is a directed graph that contains all the k-mers present in the read dataset as vertices, and all the possible (k -1)overlaps between k-mers as edges. Such graphs have been widely used in de novo assembly  (14). The idea is the following: if a dataset contains two sequences that are identical, except for one character, then these sequences generate a bubble in the graph (see an example in Figure  2 ).
Formally, a bubble in a de Bruijn graph is composed of two distinct paths of k + 2 nodes, having the start and the end nodes in common. Precisely, KISSNP2 detects couples of paths of length 2k -1, denoted by p 1 and p 2 , that correspond to polymorphic sequences, i. e. sequences of bubbles excepting the two extreme nodes. Formally, these two paths can be written as p 1 = p␣q and p 2 = p␤q, with p, q being (k -1)-mers, and ␣ and ␤ are single nucleotides such that ␣ = ␤. In KISSNP2, this idea is exploited by generating the de Bruijn graph of all input dataset(s) pooled together, and by searching the previously described bubbles in the graph. To avoid k-mers generated by sequencing errors, only kmers whose support (coverage) is above or equal to a userdefined threshold, c, are taken into account in the construction of the de Bruijn graph.
Retrieving sequence contexts. Each sequence in the multifasta file output by KISSNP2 is composed of the 2k -1 nucleotides, which correspond to the bubble, together with its left and right contigs that are extending the 2k -1 sequences. For every such contig, the length of the longest unambiguous context (longest non-branching path in the graph starting from the bubble) is also indicated. Recon-  structing these contexts is done with the MINIA assembler (15) algorithm. Its contiging algorithm was evaluated within the Assemblathon2 challenge (17) (team 'Symbiose'). KISSREADS module. One should note that it is not possible to compute read coverages, nor read quality information, based solely on a raw de Bruijn graph. Moreover, all de Bruijn graph-based methods may build chimeric sequences due to overlapping k-mers, which are never present in the same read.
These sequences are said to be non-readcoherent. The KISSREADS module is meant to filter out bubbles composed of non-read-coherent sequences, to add coverage and quality information on the remaining ones, and finally to rank SNPs (see the section below). Given a sequence s and a set of reads R mapped on s, we say that s is read-coherent if, for each position of s, at least c reads are mapped, with c a user-defined threshold. Note that reads are mapped in a semi-global manner: mapped reads may have a prefix starting before the first position of s and/or may have a suffix ending after the last position of s. By default, one substitution is authorized which roughly corresponds to current error rates. Moreover, knowing that in the DISCOSNP framework the sequence s represents one of the two alleles of a SNP, no substitution is authorized on the polymorphic position during the mapping.
Ranking SNPs DISCOSNP scores each SNP according to the coverage repartition of its alternative paths between the conditions. The aim is to rank best the SNPs that are the most discriminant between the samples. For a given SNP, the score is the Phi coefficient of the table of read counts for each path and each dataset, computed as follows: χ 2 n. It can be seen as a normalized Chi-squared statistics that varies between 0 and 1. A high score, close to 1, is obtained if the frequencies of the paths are very different between datasets, the best case being for homozygous SNPs between two datasets, where each path is strictly specific to one dataset. Notably, this score ranks poorly bubbles that are due to sequencing errors or inexact repeats as they are likely to have similar repartitions in all datasets (small frequency for an error, and equal frequency for repeats).
Moreover, the normalization prevents from over-scoring highly covered bubbles which are often due to repeats. Since this ranking favors SNPs for which one variant is enriched in one read set, it is not well suited to select SNPs that are heterozygous in all read sets. It is also not usable with only one dataset, one diploid individual or a pooled sample. As shown in the 'Results' section, other rankings can be used in these cases.
DISCOSNP input and output In summary, DISCOSNP takes as input any number of read sets, and has two main parameters k and c, respectively the k-mer size used to build the de Bruijn graph and the minimal coverage a k-mer should reach to be inserted in the graph. When applied to two read datasets or more, DISCOSNP is executed only once as all datasets are pooled together. All isolated SNPs shared by any number of samples are output as single calls. Finally, the DISCOSNP output is a sorted multi-fasta file, in which every consecutive couple of sequences corresponds to the two 2k -1 paths of a SNP, surrounded by its left and right contigs. Headers of the sequences give information about read coverage and average PHRED quality for each input dataset, lengths of unambiguous left and right extensions, and Phi coefficient of each SNP.
Simulated datasets and evaluation We propose here an overview of the evaluation approach. Supplementary details on the protocols for generating data and for evaluating the results are provided in Supplementary Additional File 1. All tools were run with at least k = 31 and a minimal coverage of 4 (k-mers that were seen less than four times were considered to be sequencing errors) or based on the developers' advices. When several parameter sets were tested, presented results were obtained with those giving the best results. Full details on parameters and full results are proposed in the Supplementary Additional File 1. Simulation and evaluation for one or two diploids. In order to evaluate the behavior of DISCOSNP on large, complex genome sequences, and to compare it to the other referencefree SNP discovery tools, we propose an experiment simulating the sequencing of two diploid, human chromosome 1 individuals.
We produced two SNP reference files, re f snps. f a (as described in 'Precision and recall computation', in Supplementary Additional File 1): one for the HG00096 individual, and a second one when considering the two individuals. Predicted SNPs were compared to one of these files (depending on the dataset(s) on which DISCOSNP was applied) in order to compute the number of true and false positives. A predicted SNP for which the two paths match exactly the two paths of a simulated SNP present in this reference file is a true positive (TP), else it is a false positive (FP). In the same way, a SNP from the reference file not matched by any predicted SNP is a false negative (FN). The recall is given by the number of TP divided by the number of TP plus the number of FN. The precision is given by the number of TP divided by the number of TP plus the number of FP.
With this strategy, while considering all the 30 individuals together, 69 600 SNP sites were generated, covering ≈1. 5% of the genome. Similarly to the diploid dataset, we simulated a 40x sequencing of each of the 30 individuals, with 100-bp reads and 0. 1% error rate. Thus, 1 855 870 reads were generated per read set. Finally, we created a reference file containing the isolated-SNPs per subsets of individuals: the first two, the first three, and so on. For each of these subsets, a SNP was considered as isolated if no other SNPs were simulated in the k -1 positions before and after the SNP's locus, inside this subset. In the presented results, for two individuals, 4164 simulated SNPs are isolated while 268 SNPs (≈6%) are not. For 30 individuals, 25 993 SNPs are isolated while 333 930 (≈92%) are not.
Mouse dataset DISCOSNP was applied on a real dataset to compare two mouse inbred strains, FVB/NJ and C57BL/6NJ, the latter corresponding to the mouse reference genome (NCBIM37). Reads were retrieved from the European Nucleotide Archive (www. ebi. ac. uk/ena/), 987 million reads for the C57BL/6NJ strain (ERP000041) and 1888 million reads for the FVB/NJ strain (ERP000687). To compare with DISCOSNP results, we retrieved the set of predicted SNPs obtained with the same data by Wong et al. (18)  (www. sanger. ac. uk/resources/mouse/genomes/, file mgp. v3. snps. rsIDdbSNPv137. vcf). We selected SNPs having distinct genotypes in FVB/NJ and C57BL/6NJ strains. This provided 5 020 489 SNPs, among which 2 472 456 (≈49. 2%) are isolated. We evaluated the number of isolated SNPs found by the two methods and those found specifically by one method or the other, by comparing the DISCOSNP output to the Wong et al.
Saccharomyces cerevisiae dataset DISCOSNP was applied on a real S. cerevisiae dataset, for which several SNPs have been biologically validated in a recent study  (19). In this study, three glucose-limited, chemostat-evolved populations of haploid S288c, named E1, E2 and E3, were sequenced every ≈70 generations, giving eight samples per population. Using a reference-based mapping approach, 110 mutations were discovered, among which only 33 have a minor allele frequency (MAF) >10% and were confirmed by Sanger sequencing. Among these 33 e11 Nucleic Acids Research, 2015, Vol. 43, No. 2 PAGE 6 OF 11 SNPs, 30 are isolated (with k = 31). Therefore, in order to assess the performance of DISCOSNP on this dataset in terms of recall, these 30 SNPs were used as a reference. The 24 read sets were downloaded from the NCBI Sequence Read Archive (with the accession number SRA054922), and processed to remove barcode and adapter sequences as in the initial study.
Tick dataset DISCOSNP was applied on real sets of reads as part of a population genetic study of the tick Ixodes ricinus  (2). DNA was extracted from two tick pools, one composed of ten individuals from Gardouch (close to Toulouse), France, and another composed of 20 individuals from Malville (close to Nantes), France. A genomic reduction was applied on these two pools by selecting the piece of an agarose gel containing DNA fragments, with a length of 500-600 nucleotides within the smear obtained following the enzymatic digestion by Msel of genomic DNA. This reduction corresponded to 3. 8% of the initial genome. The DNA was sequenced by 454 Roche pyrosequencing, leading to the generation of 1 389 201 reads in two libraries (730 482 for one and 658 719 for the second). After quality trimming (where reads or ends of reads with a PHRED quality score inferior to 20 or that did not contain the expected restriction site were deleted), a total of 996 508 reads (536 061 for the first pool and 460 447 for the second) with an average length of 529 bp were used for analysis with DISCOSNP.
As in this study framework one is not interested by SNPs discriminating the two datasets, we did not use the Phi coefficient for selecting SNPs for experimental validation. Among the 1768 SNPs meeting these criteria, 384 were randomly picked for genotyping validation. Primers were designed for each selected SNP. To validate them, we performed a genotyping run using Fluidigm technology, where primers are combined with fluorochrome (VIC or FAM for each allele)  (22). Reading the fluorescence allows to determine the genotype of the individual typed at each locus (homozygous XX or YY, heterozygous XY).
RESULTS We propose experiments that aim at (i) assessing the quality of DISCOSNP results on simulated datasets, in comparison with state-of-the-art reference-free SNP detection methods (including a hybrid approach); (ii) showing how DISCOSNP performs on real data, with biological validation. In addition, the computational resources (time and memory) required by DISCOSNP are compared with those required by the other tools. Results were obtained with DISCOSNP, version 1. 2. 1 available online, as well as with the latest versions of NIKS, BUBBLEPARSE (released on April 2013) and CORTEX (COR-TEX VAR v1. 0. 5. 21). The tests were performed on the GenOuest (genouest. org) cluster, composed by Intel Xeon C core processors with speed varying between 2 and 2. 8 GHz.
Detection of isolated SNPs from two to 30 haploid datasets simulated from E. coli We simulated 30 read sets from E. coli genomes, in which SNPs were inserted in order to mimic a realistic allele frequency spectrum of several individuals (see 'Materials and Methods' section for details). Below, we present the results obtained by different tools on (i) two haploids, and on (ii) three to 30 haploids, with respect to the set of isolated SNPs. Results on two haploids. Results of DISCOSNP applied with default parameters on 2 of the 30 haploid genomes serve as proof of concept of the approach. Results reach high precision and recall: 98. 81% of predicted SNPs are true positives, while 97. 31% of simulated isolated SNPs were recovered. Moreover, these results were obtained in 3 min 49 s, and needed no more than 7MB of RAM memory.
Results presented in Table  1  show that, on a simple dataset composed by two bacterial genomes, all tools give similar results in terms of precision/recall (except for KISSNP and NIKS, all results have precision ≈99. 19 ± 0. 66% and recall ≈96. 56 ± 1. 41%). However, DISCOSNP runs faster than other tools while being by far the one needing the smallest amount of memory, by at least three orders of magnitude. Results on more than two haploids. When applied on more than two datasets, DISCOSNP still produces high quality results, with precision reaching 99. 83%, and recall decreasing only slightly (94. 29%) for 30 individuals (Figure  5a ). Notably this shows that DISCOSNP results remain constant regardless of the number of input datasets, unlike CORTEX whose recall and precision drop for more than two datasets.