dataesr/scientific-paragraphs-categorization huggingface dataset:
sequana coverage --input virus. bed -w 4001 -o The -o option indicates that the input is a circular DNA molecule. The RM window can be tuned using the -w option. Several chromosomes may be present (e. g. , fungus case). By default, all chromosomes are analyzed, but users can select a specific one using the -c option. Other useful options are the ability to change the thresholds on the z-score, cluster close ROIs, and analyze the data by chunks (useful for large eukaryote genomes). An additional feature is the ability to download a reference genome (given its ENA  [25]  accession number). This is achieved internally using BioServices  [36] , which can switch between the ENA and National Center for Biotechnology Information web services to download the data automatically. Regions of lower genome coverage are sometimes related to repeated content or unusual GC content  [37].
Using the reference, we provide a GC content vs coverage plot in the report as shown in Fig. 6. GenBank annotations can also be downloaded to annotate ROIs. The output is a directory that contains (for each contig/chromosome): an HTML report, a summary file (JSON format), and a comma-separated values (CSV) file with detected ROIs. In addition, we provide a multiQC report  [38]  via a plugin available in the Sequana library. The multiQC report contains a summary of the mapping metrics, including the DOC and BOC metrics, the number of ROIs, and the centralness (defined in this manuscript). The CSV file is structured with one ROI per row, including information such as the location, length, mean z-score, max z-score, and mean coverage. In the individual HTML reports, JavaScript plots are provided together with the ROIs for quick inspection (not available for genomes >5 Mbp).
Example: viral genome characterization In this section we illustrate the usage of sequana coverage on the viral test case (described in the Data Description section). This 18-kb-long genome contains three SNVs (coverage of zero) of length 3, 1, and 1 base with two of them separated by only two bases; two deleted events (700 and 800 bases long); and three short depleted regions with a low signal-to-noise ratio (see Fig. 7  for a visual representation). When running sequana coverage, the default window parameter is set to 20,000 bases for genomes longer than 100,000 bases. Otherwise, the default value of W is set to a fifth of the genome length. Here, it means W ∼ 4,000. Taking into account the circularity of the genome, we obtain the results shown in Fig. 7  and Table  2 , where nine ROIs are found distributed into eight depleted regions and one enriched region.
CNV detection In extending the functionality of sequana coverage to include larger genomes, we also explored its ability to detect CNVs. CNV detection methods can be categorized into five different strategies depending on the input data: paired-end mapping, split-read, read depth (i. e. , genome coverage), de novo assembly, and combinations of these approaches. Among the numerous tools based on the genome coverage reported in  [17] , we choose CNVnator  [20] , which is able to detect CNVs in various sizes ranging from a few hundred bases to mega-bases. CN-Vnator can also handle whole genome datasets and exhibits a good precision at detecting breakpoints. We also considered a more recent tool, CNOGpro  [16] , which is dedicated to prokaryotic whole genome sequencing data. As stated in  [17] , none of the various tools have been able to detect the full spectrum of all types of CNVs with high sensitivity and specificity.
To increase the performance in detecting CNVs and reduce false positives, a combinatorial approach could take advantage of the different methods. We first examined the false-positive rate of sequana coverage on simulated data. Technical details can be found in the Notebook 5  [18]. Simulated paired-end data were used to create 100 genome coverage data for S. aureus, each one having a depth of 100X. The number of ROIs detected with sequana coverage is 17. 5 on average (standard deviation of 6). The 1,750 ROIs are plotted in Fig. 8 , showing their mean z-scores vs lengths. We observed that no z-score are above 5 (in absolute value). However, the sizes of the ROIs vary widely, up to 80 bases. Such events are not caused by genuine features in the genome (e. g. , high GC, repeats). Indeed, across the 100 independent lists of ROIs, the longest events do not appear at the same location on the reference.
CN reported for duplicated and deleted events is 1. 49 ± 0. 023 and 0. 5 ± 0. 026, respectively. The simulated data indicate that the algorithm can detect short CNVs (from 1,000 to 8,000) with high sensitivity and accurate estimate of CN and location. If we set the threshold to a mean z-score of 5 and discard events with length below 100 bases, there are no false-positive detections. For a comparison with published tools using real data, we examined the S. aureus case used in  [20]. We ran sequana coverage and CNVnator on the 3-Mbp genome. CNVnator has a parameter called bin, which is essentially used to define the breakpoint resolution accuracy. We used bin parameters of 1, 6, and 100 (default) where 6 was chosen as the optimal bin size for the sequencing depth considered (500X). Here, we refer to the instructions found in  [20]  that led to an empirical equation bin = 2,500/DOC (see also  Notebook 8 [18] ).
9 ). In terms of computational time, sequana coverage takes 1 minute to process this 3-Mbp genome, irrespective of W, while CNVnator takes about 25 minutes, 5 minutes, and 40 seconds for bin = 1, 6, and 100, respectively. Then, we looked at a population of six isolates of S. aureus used in  [16]. The six datasets have a wide range of sequencing depth: 165, 61, 36, 94, 1,100, and 34, for the isolate ERR043367, ERR043371, ERR073375, ERR043379, ERR14216, and ERR316404, respectively. We compared the results provided in the supplementary data of  [16]  with those obtained by running We focus on the region between positions 86,500 and 90,000. We analyze the data (black lines) with sequana coverage (horizontal red lines) and CNVnator (green areas). We also report the results of CNOGpro (red areas in the top panel only).
CNOGpro detects the complex event as a single event with poor breakpoint resolution (end location is offset by 300 bases); see text for an explanation. CNVnator detects one event in three isolates, two events in two isolates) and four events in one isolate (fifth row); the gap in the middle of the genomic region considered is missed in 50% of the cases; breakpoint resolution is high. sequana coverage reports four to six events; the breakpoint resolution is high; the event in the middle is systematically ignored, as it should be, given its length is about 100 bases. sequana coverage and CNVnator. In CNOGpro's supplementary, the authors report 43 CNVs with various CNs. After visual inspection, we believe that seven are false positives and the remaining are confirmed by sequana coverage. It is important to note that, unlike CNVnator and sequana coverage, which rely on the data to find the breakpoint of the ROIs, CNOGpro breakpoints are based on annotation and individual genes (or intergenic segments) assuming that duplications and deletions work at the gene level.
The longest chromosome (chr1), with 250 Mb, is analyzed in about 5-6 minutes. A Snakemake  [35]  pipeline was also recently implemented within Sequana  [34]  (named Coverage), allowing each chromosome to be analyzed independently. Using 24 cores, we could analyze the 24 chromosomes in about 7 to 8 minutes, which is basically the time needed to analyze the longest chromosome. A graphical interface using Sequanix  [39]  (a Snakemake graphical user interface) is also available, making the configuration of the parameters and execution of the analysis on a cluster accessible. With additional features such as the ability to annotate the ROIs with GenBank files and the identification of repeated regions, we believe that the stand-alone application sequana coverage will help researchers in deciphering the information contained in the genome coverage.
Indeed, one only needs to insert the next sample into the sorted block and delete the earliest sample from the sorted block. A standard Python module named bisect provides an efficient insertion in sorted data (keeping the data sorted). The complexity of this sorting algorithm is O(log W). So far, we have neglected the cost of the insertion and deletion steps, which is not negligible. For instance, in Python language, one of the most common data structures is the list. It is a dynamically sized array (i. e. , insertion and deletion of an item from the beginning or middle of the list requires moving most of the list in memory), and the look-up, insertion, and deletion have a O(n) complexity. So, the RM is actually dominated by the slow O(n) insertion and deletion steps. A better data structure is available thanks to the blist package; it is based on a so-called Btree, which is a self-balancing tree data structure that keeps data sorted.
The blist allows searches, sequential access, insertions, and deletions in O(log n) (see https://pypi. python. org/pypi/blist/ for details). Based on materials from http://code. activestate. com/recipes /576930/, we have implemented these two variants of RM functions in Python available in the Sequana  [34]  library. We also considered established numerical analysis tools from the SciPy  [45]  and Pandas  [32]  libraries. We finally compare the four implementations in terms of computation time and complexity, as shown in Fig. 12. It appears that the Pandas implementation is the fastest. For W >20,000 up to 200,000, our implementation is 2 to 3 order of magnitude faster than the SciPy version but 4 to 5 times slower than Pandas. We should emphasize the fact that the SciPy function has additional features since it is available for N-dimensional datasets, whereas we restrict ourselves to 1-D datasets.
In Sequana, the two variants only differ in the data structure being used to hold the data (list vs blist). Figure  12  shows the difference between the list and blist data structures that is marginal for low W values, while for large values asymptotic behaviors are reached, showing the interest of the blist over the list choice. We also see that our implementation with blist has a lower complexity than the Pandas implementation. However, for the range considered, Pandas is always the fastest choice.
Figure10: Detection and segmentation of complex events in a population sample. We focus on the region between positions 86,500 and 90,000. We analyze the data (black lines) with sequana coverage (horizontal red lines) and CNVnator (green areas). We also report the results of CNOGpro (red areas in the top panel only). CNOGpro detects the complex event as a single event with poor breakpoint resolution (end location is offset by 300 bases); see text for an explanation. CNVnator detects one event in three isolates, two events in two isolates) and four events in one isolate (fifth row); the gap in the middle of the genomic region considered is missed in 50% of the cases; breakpoint resolution is high. sequana coverage reports four to six events; the breakpoint resolution is high; the event in the middle is systematically ignored, as it should be, given its length is about 100 bases.
Acknowledgements We are grateful to  Nicolas Escriou  ( Institut Pasteur ) for providing the FastQ and reference of the virus test case. We are also grateful to  Benoit Arcangioli (Institut Pasteur) and Serge Gangloff (Institut Pasteur)  for providing the FastQ files and reference of the S. pombe test case. We thank  Juliana Pipoli da Fonseca  for her various comments on the manuscript. We are also grateful to the reviewers who suggested the CNV studies.
Availability of source code r Project name: Sequana (sequana coverage standalone), ver- sion 0. 7. 0 r Project home page: http://sequana. readthedocs. org r Operating system(s): Platform independent r Programming language: Python 3 r Containers: Sequana is available on Bioconda channel  [40, 41] , and we also provide a Singularity container  [42]  (version 0. 7. 0). See http://sequana. readthedocs. org for details. r License: BSD 3-clause Revised License.
Availability of supporting data The datasets supporting the results as well as additional files used to created them are available within a Synapse project  [29]. More specifically, the BED files mentioned in the Data Description section corresponding to the virus, bacteria, and fungus are available under: doi:10. 7303/syn10638370. 1 (JB409847. filtered. bed), doi:10. 7303/syn10638494. 1 (JB409847. filtered. bed), and doi:10. 7303/syn10638487. 1 (S pombe. filtered. bed), respectively. In addition, we provide the genome reference used in Fig. 6  (doi:10. 7303/syn10638477. 1). The datasets are also available on a GitHub repository  [18]  together with a notebook that reproduces the figures. Finally, note that the BED files can be recreated using the original FastQ files available on doi:10. 7303/syn10638358.
Background This data paper provides a description of OpenNahele, the open Hawaiian forest plot database. OpenNahele includes 530 forest plots across the Hawaiian archipelago containing 43,590 individuals of 185 native and alien tree, shrub and tree fern species across six islands. We include estimates of maximum plant size (  D95  and D  ) for 58 woody plant species, a key functional trait associated with dispersal distance and competition for light. OpenNahele can serve as a platform to test key ecological, evolutionary and conservation questions in a hotspot archipelago.
General description Purpose: To facilitate the analysis of biodiversity patterns within and across islands in Hawaiian forests, we present the OpenNahele database ('nahele' means forest in Hawaiian). This database compiles forest plot data from all six major islands of the Hawaiian archipelago and contains 530 plots, 185 tree, shrub and tree fern species and 43,590 individuals (Tables  1, 2 ). The size of each individual (diameter at 1. 3 cm; DBH) is included in the database. For each species, we also include information about whether the species is native or alien to the Hawaiian archipelago and whether the species is cultivated or not. We harmonised taxonomic names, species abundance and individual size from multiple studies to facilitate the calculation of diversity metrics that are comparable.
Data Source Islands (#) Plots (#) Plot size (m ) Min. DBH (cm) Individuals (#) Species (#)   1. Basic description of forest plot data for each data source in OpenNahele. Table  2. Sampled area, number of plots and number of individuals and species per island in OpenNahele. Table  3. Relative abundance (proportion of individuals) of the 25 most abundant tree, shrub and tree fern species across the Hawaiian archipelago in OpenNahele. The OpenNahele database can be used to examine cross-scale biodiversity patterns and drivers of and threats to biodiversity across the Hawaiian archipelago. This database provides unprecedented geographic coverage across Hawaiian forests (Fig. 1 ). While this database captures only a fraction of the entire Hawaiian flora, it provides a realistic snapshot of the current state of Hawaiian forests.
For example, this database shows that only a few native and alien species are dominant in Hawaiian forests, such as Metrosideros polymorpha, a native tree in the Myrtaceae family (Fig. 2 ; Table  3 ). The database also reveals that alien invasions are widespread across Hawaiian forests, having occurred on all major islands and in 45% of plots (Table  4 ). While alien species represent 11. 7 % of individuals in the database, some species are highly abundant in the plots where they occur. The median abundance of stems of alien species in invaded plots is 44. 5%, but varies markedly across islands from 5% on Lana'i to 87% on O'ahu (Table  4 ). As sampling effort and plot size varies within the database, we provide relative measures of abundance that are standardised to a common area (see Sampling methods) to examine community structure and caution against using raw measures of abundance, as they may introduce bias to diversity indices that are sensitive to the number of individuals.
The OpenNahele database also can be used to explore ecological differences amongst species. For example, adult plant size can be used to assess the extent to which species' geographic ranges are related to dispersal or if they are limited by habitat availability (Fig. 3 ;  Gaston and Fuller 2009). The OpenNahele database will be maintained and curated as data from future censuses and new studies become available to capture temporal dynamics of populations and communities across Hawaiian forests.
Sampling methods Study extent: To compile a database of plots in Hawaiian forests (Suppl. material 1), we consulted local experts with extensive experience in forestry, community ecology and botany, as well as experts with knowledge of former and ongoing research projects in Hawai'i, USA. From an initial list of publicly available sources and published studies, we examined each to determine if they met our inclusion criteria. Our inclusion criteria were that each study reports for each plot: i) geographical location, ii) species identity and iii) abundance as the number of individuals of trees, shrubs, and tree ferns. We downloaded raw data or obtained it directly from data owners, which also included individual size either as diameter at 1. 3 m (diameter at breast height; DBH) or in size classes.
In total, we identified 6 unique studies meeting our inclusion criteria that together comprise 43,590 individuals of 185 species within 530 plots across six islands (Table  1;   Sampling description: Studies in the OpenNahele database used different plot sizes and minimum size thresholds (Table  1 ). Plot sizes ranged from 12. 97 m to 40,000 m and the median plot size across the database is 1,000 m. Half of the studies used one minimum size threshold and included all individuals above that in their inventory. The other half of the studies used a nested sampling approach, whereby smaller subplots were placed within each plot to assess individuals below the size threshold of the full plot. The minimum size threshold varied across studies from 1 to 2. 54 cm DBH. Currently, all studies have only conducted one inventory.
Geographic coordinates of all forest plots were converted to the WGS84 coordinate system, a standard coordinate system with a spheroidal reference surface, to facilitate the retrieval of climate, topographical and geological data. Locations of plots in the USFS FIA were fuzzed up to 1. 6 km of their exact locations (US Forest Service 2016), but those of other studies were not altered. Geographic coordinates were checked visually. Most studies inventoried trees, shrubs and tree ferns. While tree ferns do not have true wood, they play an important ecological role in Hawaiian forests  (Zimmerman et al. 2008 ). One study, Knight & Barton, did not record the presence of tree ferns (e. g. Cibotium spp. , Sadleria spp. ). However, only two plots in that study were located in areas where tree ferns occur and there are many other nearby plots in the database that included tree ferns.
As not all studies included lianas, i. e. woody vines, we removed species classified as such by USDA Plants (USDA NRCS 2018). Data from one study within the OpenNahele database, HIPPNET (Ostertag et al. (  2014 )), are curated on an ongoing basis and expanded with data from subsequent censuses; interested data users should contact HIPPNET via the CTFS-ForestGEO website.
Abundance To facilitate aggregation of abundance data across studies that differ in plot size, we calculated abundance of individuals per species on a per-hectare basis: Abundance per ha = Abundance / Area x 10,000 where abundance is the number of individuals per species and Area is the plot (or sub-plot) area in square metres. Individual size was converted to centimetres if measured as DBH or classified as greater than or less than 5 cm DBH if individual size was not measured. As data sources used different minimum DBH thresholds, which may influence the number of individuals in a plot in a systematic way, i. e. plots with larger DBH thresholds will have fewer individuals than those with smaller DBH thresholds and, therefore, species diversity estimates, we removed individuals smaller than 5 cm DBH.
Maximum plant size We estimated maximum plant size for individual species in two ways: as the 95th percentile of stem diameter of all diameters > 0. 1 x maximum observed diameter (D95 ) and as the mean diameter of the three largest individuals across the database (D ;  King et al. 2006). In total, we estimated both measures of maximum plant size for 58 woody plant species that had at least 20 individuals = 5 cm DBH (Suppl. material 2). We compared both measures using Spearman's correlation coefficient and found that while strongly and positively correlated (rho = 0. 89, p-value< 0. 001), D was on average 49% larger than D95. As D95 is not sensitive to sample size  (King et al. 2006) , we recommend using this measure over D , particularly for studies using it as a functional trait in combination with species abundance data.
Quality control: Taxonomic names were resolved and harmonised with The Plant List v. 1. 1 (The Plant List 2013) using the 'Taxonstand' package  (Cayuela et al. 2017). Family names and orders were also retrieved and used to identify angiosperms and monocots following the Angiosperm Phylogeny Group III (Angiosperm Phylogeny Group 2009). Native status was obtained by consulting the electronic Flora of Hawaii  (Wagner et al. 2005). Individuals not identified to the species level were classified as 'uncertain', unless the genus was endemic to Hawai'i. As not all introduced species have been naturalised, i. e. they are cultivated in forestry plantations or as ornamentals but have yet to establish selfsustaining populations, we also obtained the cultivation status for introduced species using the Pacific Island Ecosystems at Risk database (US Forest Service 2017).
Geographic coverage Description: The 530 plots in the OpenNahele database are located on all six major islands of the Hawaiian archipelago (Fig. 1 ). Forested areas are well covered by plots on most islands and include a wide range of habitat types, from tropical dry forests to subalpine shrublands. However, not all islands were sampled with the same intensity (Table  2 ); 380 plots are located on the largest island, Hawai'i, and the other, smaller islands have between 5 and 59 plots. One potential limitation with this database is that most studies, with the exception of those collected by the USFS FIA, did not locate plots randomly, a fact which may introduce a bias towards forests with relatively low amounts of alien species. Coordinates:  18. 90986 and 22. 23583 Latitude; -154. 8058 and -160.
Taxonomic coverage Description: In total, the OpenNahele database contains 185 tree, shrub and tree fern species, of which 61% and 39% are native and alien, respectively, and which represent 16% of the 1,155 woody species that occur across the Hawaiian archipelago  (Wagner et al. 2005). The database captures a relatively small proportion of woody species in Hawai'i, possibly because many species may not reach the minimum size limit (5 cm DBH) and that plots were only located in forests and not in ecosystems where woody species occur but are small in stature or are not dominant, e. g. shrublands and grasslands. However, the database has a similar proportion of alien woody species (39%) as found across the entire Hawaiian archipelago (41%;  Wagner et al. 2005).
Dominant species Metrosideros polymorpha is hyperdominant in Hawaiian forests and represents 33% of all individuals greater than 5 cm DBH (Fig. 2 ;Table  3 ). Two species of native tree ferns, Cibotium glaucum and C. menziesii, are amongst the five most abundant species in the database. Alien species have heavily invaded Hawaiian forests; fifteen of the 25 most abundant species in the database are alien (Table  3 ).