dataesr/scientific-paragraphs-categorization huggingface dataset:
Construction of Core Subsets To compare the performance of current state-of-the-art methods to construct core subsets, as a benchmark, we estimated the minimum size necessary to capture all the observed alleles using the MSTRAT program (Figure  1 ). The size assessment indicated that 80 entries were necessary to capture the total allelic diversity (16% of OWGB Marrakech). Then, at this sample size, four different sampling methods were first tested: 1. The maximizing method (M method) implemented in the MSTRAT program. By using an iterative maximization procedure, MSTRAT examines all possible core subsets and singles out those that maximize the number of alleles (and/or trait classes) in dataset for one sample size. The program allows to specify a compulsory set of accessions, called a ''kernel'', that will always be included in the core subset.
In this case, maximization was focused on complementing alleles not included in the kernel. The Shannon-Weaver diversity index  [73]  was used as a second criterion to classify core subsets capturing the same number of alleles. A previous analysis revealed that when a weight of 60% was assigned to D CE and 40% to Cv, all observed alleles were captured in the sampled subset (Figure  S1 ). 3. The maximum length sub-tree method (MLST method) implemented in the DARWIN v. 5. 0. 137 program  [41]. Starting from a diversity tree, the procedure is performed step by step. At each step, the unit for each pair with the minimal length of the external edge in the tree is removed. The procedure searches for the most unstructured tree, i. e. a star-like tree, by successive pruning of redundant units. The genetic distance between genotypes was calculated using the sample matching coefficient  [76]  and the tree was drawn based on the Neighborjoining method  [77].
Assessment of Core Collections for Association Mapping Purposes As the sub-structure within subsets and the relatedness between genotypes (known also as the kinship coefficient) are the major components to take into consideration in association mapping analyses  [80] [81] [82] , an assessment of both factors in proposed core collections was performed. Two approaches were used to assess the genetic structure; (i) principal coordinate analysis (PCoA) implemented in the DARWIN v. 5. 0. 137 program using a simple matching coefficient to describe the spatial distribution of genotypes; and (ii) model-based Bayesian clustering implemented in STRUCTURE v. 2. 2  [83]  according to the parameters described in Haouane et al. [52]. The reliability of the number of K clusters was checked using the ad-hoc DK measure  [84]  with the R program whereas the similarity index between 10 replicates for the same K clusters (H9) was calculated via CLUMPP  [85].
The redundancy function of MSTRAT program thus revealed that 92 entries (CC 92 ) were necessary to capture all 279 alleles. As for CC 94 , 72 genotypes were common between all 200 constructed core collections of 92 entries (result not shown), whereas a panel of 91 genotypes could be used to select a combination of 20 complement genotypes to capture the total allelic diversity. One core collection of 92 entries among 200 was arbitrary chosen and compared to the above described CC 94. The results indicated that 99% of the trait classes (211 among 213) were captured in this core collection and similar values were obtained regarding D CE , Sh and He for both core collections (Table  2 ). In addition, 85 genotypes  were shared between CC 92 and CC  94. Hence, phenotypic data may have a limited effect since similar results were obtained regardless of the sampling method used, i.
Further studies using other sets of molecular markers (e. g. SNP) could confirm our assumption. Furthermore, despite the fact that maternal lineage polymorphism of is lower within olive varieties than noted in olive oleasters  [67] , therefore chloroplast sequence information is substantial when establishing core collections. This information optimises sampling to clarify the evolutionary history of olive varieties and therefore their involvement in agronomic traits of interest alone or in association with nuclear genes. Otherwise, the compiled phenotypic data was used with caution in the present study since not all varieties were completely characterized with the 72 agro-morphological traits and phenotypic data was gathered from different olive databases according to the variety names  [68] [69] [70] [71] [72].
Acknowledgments The authors would like to thank  X. Perrier  and  B. Gouesnard  for their kind remarks on the earlier version of the manuscript,  S. Santoni  and  Ch. Tollon  for their kind support in the molecular analysis, and  M. H. Muller  for comments on the final version of the manuscript. They also acknowledge the  International Olive Oil Council  and  INRA Morocco  for their contribution in the founding and management of OWGB Marrakech. AEB is a PhD student who will defend his thesis entitled ''Sampling methods to establish olive core collections for association mapping studies'' at  Ghent University, Belgium , in 2013. He conducted his research work at  Montpellier SupAgro , UMR AGAP in the framework of a thesis study agreement between  Ghent University  and  Montpellier SupAgro.
Background The de Bruijn graph of a set of DNA or RNA sequences is a data structure which plays an increasingly important role in next-generation sequencing applications. It was first introduced to perform de novo assembly of DNA sequences  [1]. It has recently been used in a wider set of applications: de novo mRNA  [2]  and metagenome  [3]  assembly, genomic variants detection  [4, 5]  and de novo alternative splicing calling  [6]. However, an important practical issue of this structure is its high memory footprint for large organisms. For instance, the straightforward encoding of the de Bruijn graph for the human genome (n ≈ 2. 4 • 10 9 , k-mer size k = 27) requires 15 GB (n • k/4 bytes) of memory to store the nodes sequences alone. Graphs for much larger genomes and metagenomes cannot be constructed on a typical lab cluster, because of the prohibitive memory usage.
Recent research on de Bruijn graphs has been targeted on designing more lightweight data structures. Li et al. pioneered minimum-information de Bruijn graphs, by not recording read locations and paired-end information  [7]. *Correspondence: chikhi@irisa. fr 1 Computer Science department, ENS Cachan / IRISA / INRIA, Rennes 35042, France Full list of author information is available at the end of the article Simpson et al. implemented a distributed de Bruijn graph to reduce the memory usage per node  [8]. Conway and Bromage applied sparse bit array structures to store an implicit, immutable graph representation  [9]. Targeted methods compute local assemblies around sequences of interest, using negligible memory, with greedy extensions  [10]  or portions of the de Bruijn graph  [11]. Ye et al.
In the first Section, the notions of de Bruijn graphs and Bloom filters are formally defined. Section "Removing critical false positives" describes our scheme for exactly encoding the de Bruijn graph using a Bloom filter. Section "Additional marking structure for graph traversal" presents a solution for traversing our representation of the de Bruijn graph. Section "Results and discussion" presents two experimental results: (i) an evaluation of the usefulness of removing false positives and (ii) an assembly of a real human dataset using an implementation of the structure. A comparison is made with other recent assemblers based on de Bruijn graphs.
Implementation The de Bruijn graph structure described in this article was implemented in a new de novo assembly software: Minia a. An important preliminary step is to retrieve the list of distinct k-mers that appear in the reads, i. e. true graph nodes. To discard likely sequencing errors, only the k-mers which appear at least d times are kept (solid k-mers). We experimentally set d to 3. Classical methods that retrieve solid k-mers are based on hash tables  [18] , and their memory usage scale linearly with the number of distinct k-mers. To deal with reverse-complementation, k-mers are identified to their reverse-complements.
k-mer counting To avoid using more memory than the whole structure, we implemented a novel, constant-memory k-mer counting procedure. The multi-set of all k-mers present in the reads is partitioned and partitions are saved to disk. Then, each partition is separately loaded in memory in a temporary hash table. The k-mer counts are returned by traversing each hash table. Low-abundance k-mers are filtered. This approach permits to count all k-mers of a human genome dataset using only a fixed amount of memory and disk space. The algorithm is explicitly described and evaluated in another article  [19].
The breadth limit prevents combinatorial blowup. Note that pairedend reads information is not taken into account in this traversal. In a typical assembly pipeline (e. g. [8] ), a separate program (scaffolder) can be used to link contigs using pairing information. Also, a gap-filling step (e. g. Gap-Closer in SOAPdenovo  [7] ) is typically used to fill the gaps between contigs in scaffolds.
On the usefulness of removing critical false positives To test whether the combination of the Bloom filter and the cFP structure offers an advantage over a plain probabilistic de Bruijn graph, we compared both structures in terms of memory usage and assembly consistency. We retrieved 20 million E. coli short reads from the Short Read Archive (SRX000429), and discarded pairing information. Using this dataset, we constructed the probabilistic de Bruijn graph, the cFP structure, and marking structure, for various Bloom filter sizes (ranging from 5 to 19 bits per k-mer) and k = 23 (yielding 4. 7 M solid k-mers). We measured the memory usage of both structures. For each, we performed an assembly using Minia with exactly the same traversal procedure. The assemblies were compared to a reference assembly (using MUMmer), made with an exact graph.
Even for large sizes, the probabilistic graph assemblies differ by more than 3 kbp to the reference assembly. We observed that the majority of these differences were due to missing regions in the probabilistic graph assemblies. This is likely caused by extra branching, which shortens the lengths of some contigs (contigs shorter than 100 bp are discarded). Below ≈ 9 bits per k-mer, probabilistic graph assemblies significantly deteriorate. This is consistent with another article  [13] , which observed that when the false positive Figure  3  Data structure sizes the probabilistic de Bruijn graph. Data structure sizes (Bloom filter, marking structure, and cFP if applicable) for the probabilistic de Bruijn graph with (top right) and without the cFP structure (top left), for an actual dataset (E.
Contigs shorter than 100 bp were discarded. Assemblies were made without any pairing information. nucleotides (94. 6%) were contained in contigs having a full-length alignment to the reference, with at least 98% sequence identity. For comparison, 94. 2% of the contigs assembled by ABySS aligned full-length to the reference with 95% identity  [8]. To evaluate another recent assembler, SparseAssembler  [12] , the authors assembled another dataset (NA12878), using much larger effective k values. SparseAssembler stores an approximation of the de Bruijn graph, which can be compared to a classical graph for k = k + g, where g is the sparseness factor. The reported assembly of the NA12878 individual by SparseAssembler (k + g = 56) has a N50 value of 2. 1 kbp and was assembled using 26 GB of memory, in a day.
As an attempt to perform a fair comparison, we increased the value of k from 27 to 51 for the assembly done in Table  2  (k = 56 showed worse contiguity). The N50 obtained by Minia (2. 0 kbp) was computed with respect to the size of SparseAssembler assembly. Minia assembled this dataset using 6. 1 GB of memory in 27 h, a 4. 2× memory improvement compared to SparseAssembler.
Chromosome 14 of the human genome In order to evaluate the quality of the results produced by Minia more accurately, we assembled the human chromosome 14 (88 Mbp ungapped) separately. The Illumina dataset is from the GAGE benchmark  [21]  (100 bp reads, all short paired-end libraries). Pairing information was not used in Minia. To establish a fair comparison with Minia, we selected two assemblies from GAGE (made with ABySS and Velvet, downloaded from the GAGE website) for which contigs were constructed without using pairing information. All the assemblies were done with a k-mer size of 31, as chosen by the authors of GAGE. Additionnaly, we executed Minia with k = 47, as this k-mer size was experimentally found to provide better results than k = 31. http://www. almob. org/content/8/1/22 QUAST v1.
3 was executed with the --gage option to evaluate the quality of the contigs of each assembly. Table  3  several contiguity, coverage quality metrics computed using the same reference genome and QUAST command-line for the three assemblies. The large misassemblies row shows the number of positions in the assembled contigs for which the left and right flanking sequences align either over 1 kbp away from each other, or on different strands or chromosomes. The local misassemblies row shows the number of positions in the assembled contigs for which the left and right flanking sequences are distant from each other by 90 bp to 1 kbp. On this dataset, Minia with k = 31 produced an assembly of lower contiguity and better accuracy to ABySS and Velvet. Minia with k = 47 produced a better assembler over all metrics than the other assemblers run with k = 31.
Simultaneously and independently of the present work, Bowe et al. [23]  have proposed a de Bruijn graph representation using 4+o(1) bits per edge. Their data structure is more succinct than ours, and at the same time more complex and difficult to implement. Since their structure has not been fully implemented yet, it is not possible to compare the practical performance with Minia. However, this new advance is very interesting and raises the question of whether their structure is the most succinct way to represent a de Bruijn graph.
The ranking is based on "the number of skilled engineers world-wide, courses, and third-party vendors". 2 The rank is computed by performing searches on a number of search engines including Google, Bing, Yahoo!, Wikipedia, Amazon, YouTube, and Baidu. The data used by TIOBE is not freely available and we could not find a description of the exact methodology for the ranking on TIOBE's site. Different from TIOBE's ranking, we investigate the popularity, interoperability, and impact of programming languages from a different angle: we do not look into search engines or analyze skilled engineers, courses, or third party vendors; rather, we analyze a hundred thousands of open source software projects. We believe that analyzing actively developed software artifacts is a promising way to understand how programming languages are being used.
We describe our approach in detail in this paper and make our data publicly available 3. Conway has also provided a ranking of languages based only on their appearance in GitHub projects  [2]. In this study we further investigate the amount of lines of code written for each language. Our study is made possible by the availability of a large amount of data in GitHub. Millions of projects are publicly available for download in GitHub. There are small and large projects; large ones include the Linux kernel (with over 10,000,000 lines of code written by more than 9,000 contributors). The availability of a wealth of project data allows us to explore a variety of interesting questions on the popularity and impact of various programming languages. We are however aware that the findings based on these open source projects are only meant to shed light, to some extent, on the usage of common programming languages.
Some languages might be under-represented in the dataset of projects because of the constraints in their development context. For example, the VHDL language, which is seldom in our dataset, is widespread for embedded systems in the industry, a domain that tend not to publish their work to open source platforms. Aside from such cases, our study provides a clear picture on the popularity of programming languages in public domain. For this study, we have obtained 100,000 git repositories from GitHub. We have analyzed the contents of each of the git repositories to find software code. We count the numbers of lines of code written in various programming languages. Based on this information, we can answer our formulated 2 http://www. tiobe. com/index. php/content/paperinfo/tpci as of June 2012 3 Upon request.
questions related to the popularity and interoperability of programming languages. We also collect various information related to project success, reported issues, and team sizes. The contributions of this work are as follows: 1) As far as we know, we are the first to analyze the popularity of various programming languages by analyzing a hundred thousands of open source software projects. We report the popularity of various programming languages in various dimensions: the numbers of projects written in a language, the numbers of lines of code written in a language, and the numbers of developers that "read"/"write" a language. 2) We analyze the interoperability of various programming languages based on projects that are developed using more than one programming language. We measure how close a pair of programming languages are based on their interoperability.
3) We describe the correlation between programming languages and project "success" in the developer community, reported bugs, and development team sizes. The rest of this paper is organized as follows. In Section II, we provide preliminary information on various programming languages and GitHub. In Section III, we elaborate the methodology of our empirical study. In Section IV, we present the results of our study. We list the threats to validity in Section V. Section VI discusses related work. We conclude and describe future work in Section VII.
A. Programming Languages Programming languages define the grammar and semantics used by human beings to communicate and interact with machines. Programming languages, such as the one used by Ada Lovelace in her first program  [5] , have existed before the invention of modern computers in the 1940s. With the advent of the computer, programmers started to use assembly languages. Seventy years later, hundreds of programming languages have been invented. While some have been discarded with the arrival of their better alternatives, others have kept their usage trends. A few more recent programming languages have quickly gained momentum in the last years. Each programming language belongs to a category that is more or less suitable for different programming tasks and computing environments. We only focus on 30 commonly known programming languages that appear in our dataset.
B. GitHub GitHub is a project hosting site that has introduced the concept of social coding by providing many developerfriendly features. It can be viewed as a socio-technical network. GitHub has thus gained a wide acceptance and adoption among developers, and its website indicates that it is currently 4  hosting more than 3,000,000 project repositories. Thanks to GitHub's extensive REST  [4]  APIs  5  , we are able to identify about 1,300,000 million repositories whose contents are publicly available for use. GitHub uses git  [7]  as its revision control system for source code versioning. It enables forking, which allows to create copies of repositories, and it supports pull requests for merging changes from copies to the source repository, to facilitate contributions from developers who are not project team members.
Aside from source code revision control, GitHub integrates common software development facilities, such as bug/issue trackers and wiki pages. A large number of projects, such as those from the Apache community, which are developed outside the GitHub platform, i. e. , autonomously or on other platforms such as Sourceforge, have their repositories mirrored in GitHub. This fact indicates that by using GitHub projects, we do not limit ourselves to the GitHub community or to git. Furthermore, a number of famous companies such as Facebook TM host open source versions of their projects on GitHub. This suggests that our project dataset contains projects managed by diverse communities. While the large number of projects hosted by GitHub and its extensive APIs for accessing the projects enable large studies like ours, there are still some challenges that we need to address, such as (1) collecting and managing huge amount of data, and (2) inferring important information which are not directly available from the APIs, such as the number of source lines of code in a project.
III. METHODOLOGY To perform our empirical study, we need a large number of projects that may represent the universe of all projects. The software projects should be written for various purposes by diverse teams in a variety of languages. We have done a manual exploratory survey of the projects on GitHub, investigating the development teams, the application domains and the range of programming languages used in the projects. We have thus found that projects hosted by GitHub are very diverse and thus suitable to the requirements of our study. We have also found that GitHub is used as the main development platform for over a million projects, and that it may also be used as a mirror for popular projects, such as Apache or Linux. For our study, we consider the first 100,000 projects returned by the GitHub APIs.
There appears to be no distinct ordering scheme in the returned list of projects  6  which also vary on subsequent requests. Given these projects, we extract several types of information for each project: General project information: Each repository is monitored by GitHub and tracked based on different features. In this paper, we consider the concepts of watchers and forks. "Watchers" is a metric for measuring interest and activity in a project. It gives an indication of the amount of attention that is given to a project by the developer community: an aspect of project "success". Watchers typically use project releases, report bugs, and incidentally promote the project in their socio-technical network. "Forks" is a metric for measuring the active involvement of the developer community in the growth of a project's code base and the improvement of its quality.
While these metrics are not absolute, they provide good insights on the "success" of a project. Source Lines of code: To compute the number of physical source lines of code (sloc), we download each project's code repository and rely on the SLOCCount 7 tool. This tool computes actual lines of code, ignoring code comments and blank lines. It supports 29 of the languages considered in our study. We have extended SLOCCount to add support for the 30th language, namely JavaScript. Developer contributions: We also consider the contributors for a given project inferred from the commits. Whether a contributor to a project is registered or not in GitHub, his information is always available in the git repositories that he contributes to. This inference is particularly important for projects that are just mirrored in GitHub but whose development is done on the project's own website.