dataesr/scientific-paragraphs-categorization huggingface dataset:
To overcome these limitations, we developed instaGRAAL, an enhanced, open-source program optimized to reduce the computational load of chromosome scaffolding and that includes a misassembly "correction" module installed alongside the scaffolder. Moreover, instaGRAAL can optionally exploit available genetic linkage data. We applied instaGRAAL to three genomes of increasing size: in the first two runs, and in order to demonstrate its added value, we applied the program to the 214-Mb and 500-Mb haploid genomes of the brown alga Ectocarpus sp. [21, 22]  and Desmarestia herbacea (unpublished), respectively. Brown algae are a group of complex multicellular eukaryotes that have been evolving independently from animal and land plants for more than a billion years. Ectocarpus sp. was the first species within the brown algal group to be sequenced (reference v1 assembly  [22] ), as a model organism to investigate multiple aspects of brown algal biology including the acquisition of multicellularity, sex determination, life cycle regulation, and adaptation to the intertidal  [22] [23] [24] [25].
A range of genetic and genomic resources have also been established for Ectocarpus sp. including a dense genetic map generated with 3588 SNP markers (v2 assembly)  [26] , which was used to comprehensively validate both a GRAAL (v3) and the instaGRAAL (v4) assemblies. In a third run, we benchmarked instaGRAAL using the human genome, to confirm that our software readily scales to larger (Gb-sized) and more complex assemblies, an important requirement to tackle the next era of assembly projects.
insta-GRAAL (https://github. com/koszullab/instaGRAAL) addresses all these shortcomings. First, we rewrote the memory-critical parts of the program, such as permutation sampling and likelihood calculation, so that they are computed using sparse contact maps. We reduced the software's dependency footprint and added detailed documentation, deployment scripts, and containers to ease its installation. Finally, we opened up multiple hard-coded parameters to give more control for end-users while improving the documentation on each of them and selecting relevant default parameters that can be implemented for a wide range of applications (see options online and the "Discussion" section). Overall, these upgrades result in a program that is lighter in resources, more flexible, and more user-friendly.
Other problems encountered with the original GRAAL program included (1) the presence of potential artifacts introduced by the permutation sampler, such as spurious permutations (e. g. local inversions) or incorrect junctions between bins; (2) difficulties with the correct integration of other types of data such as long reads; and (3) difficulties with handling sequences that were either too short, highly repeated, or with low coverage. We addressed these points by identifying and putting aside these problematic sequences during a filtering step. These sequences are subsequently reinserted into the final scaffolds, whenever possible (see the "Material and methods" section), with the help of linkage data when available. Overall, when compared to the raw GRAAL output, the resulting "corrected" instaGRAAL assemblies were significantly more complete and more faithful to the actual chromosome structure.
genome into chromosome-level scaffolds. As the supplementary movie suggests, scaffold-level convergence is visible after only a few cycles, indicating that instaGRAAL is able to quickly determine the global genome structure most likely to fit the contact data. The remainder of the cycles is devoted to intra-chromosomal refinement. Correcting the chromosome-level instaGRAAL assembly of the Ectocarpus sp. genome instaGRAAL also includes a number of procedures that aim to correct some of the modifications introduced into the input contigs from the original assembly by the Hi-C scaffolding (Fig. 4 ). We implemented it as a separate "correction" module that is automatically installed alongside the scaffolder. These modifications principally involve discrete inversions or insertions of DNA segments (typically corresponding to single bins or RFs) (see also  [13] ).
These simple steps alleviated artificial truncations of input contigs observed with the original GRAAL program. Some filtered bins had no reliable region to be associated with post-scaffolding, because their initial input contig had been completely filtered before scaffolding. These sequences, which were left as-is and appended at the end of the genome, were included into 543 scaffolds spanning 3,141,370 bp, i. e. , < 2% of the total DNA. Together, these steps removed all the misassemblies detected by QUAST. To further validate the assembly, we exploited an assembly generated by combining genetic recombination data and the Sanger assembly  [21, 26]  ("linkage group [LG] v2 assembly") as well as an assembly generated by running the original GRAAL program on the original reference v1 genome assembly ("GRAAL v3 assembly").
We searched for potential translocations between scaffold extremities between the linkage group v2 assembly and the v3 or v4 assemblies. This comparison, which was implemented as a separate module installed alongside the scaffolder, detected such events in the uncorrected v3 GRAAL assembly but none in the corrected v4 instaGRAAL assembly. The corrected instaGRAAL v4 assembly is therefore fully consistent with the genetic recombination map data, confirming the efficiency of the approach.
Comparisons with previous Ectocarpus sp. assemblies and validation of the instaGRAAL assembly We compared the corrected instaGRAAL v4 assembly with the three earlier assemblies of the Ectocarpus sp. genome mentioned above (Table  1  and Additional file 1: Table  Fig. 4  Step-by-step correction procedure. Correction procedure (top to bottom): (i) in silico restriction of the genome and binning, yielding a set of bins; (ii) reordering of all bins into scaffolds without taking into account their input contig of origin; typically, groups of bins from the same input contig naturally aggregate, but some bins get scattered to other scaffolds (e. g. , bin 13, pink arrow), while others will be "flipped" with respect to the original assembly (e. g. , bin 4, red arrows); (iii) reconstruction of the original input contigs by relocating scattered bins next to the biggest bin group; and (iv) bins in the original input contigs are oriented according to their original consensus orientation S2): (1) the original v1 genome assembly generated using Sanger sequencing data  [22] , which was assumed to be highly accurate but fragmented (1561 scaffolds); (2) the linkage group [LG] v2 assembly; and (3) the original GRAAL program v3 assembly.
We aligned the corrected instaGRAAL (v4), LG (v2), and GRAAL (v3) assemblies onto the original v1 assembly to detect misassemblies and determine whether the genome annotations (362,919 features) were conserved. We then validated each assembly using genetic linkage data (see the "Material and methods" section). For each assembly, we computed the following metrics: the number of misassemblies, ortholog completeness, and cumulative length/Nx distributions (Table  1 ). These assessments were carried out using BUSCO  [29]  for ortholog completeness (Additional file 1: Fig. S1 ) and QUAST-LG's validation pipeline  [30]  to search for misassemblies introduced in the scaffolds. QUAST-LG is an updated version of the traditional QUAST pipeline specifically designed for large genomes and is a state-of-the-art software for assembly evaluation and comparison.
99. 4% of the sequences in the 1018 contigs were integrated into the 27 largest scaffolds after instaGRAAL processing. Overall, the analysis indicated that many of the rearrangements found in the LG v2 assembly were potentially errors and that both GRAAL and instaGRAAL were efficient at placing large regions where they belong in the genome, albeit less accurately for GRAAL and in the absence of correction. These statistics underline the importance of the post-scaffolding correction steps and the usefulness of a program that automates these steps.
Comparisons with existing methods To date, only a limited number of Hi-C-based scaffolding programs are publicly available, and as far as we can tell, no detailed comparison has been performed between the existing programs to assess their respective qualities and drawbacks. In an attempt to benchmark instaGRAAL, we ran SALSA2  [33]  and 3D-DNA on the same Ectocarpus sp. v1 and Desmarestia herbacea reference genome and Hi-C reads. 3D-DNA is a scaffolder that was hallmarked with the assembly of Aedes aegypti, and SALSA2 is a recent program with a promising approach that directly integrates Hi-C weights into the assembly graph. For Ectocarpus sp. , SALSA2 ran for nine iterations and yielded 1042 scaffolds, with an N50 of 6,552,506 (L50 = 11). Its BUSCO completeness was 77. 6%, a level identical to that obtained with instaGRAAL.
Scaffolding the human genome To confirm that instaGRAAL scaffolds larger (Gb scale) genomes in a reasonable time, we ran it on the GRCh38 human genome sliced into 300-kb segments (artificial assembly), using a Hi-C dataset generated with an Arima Genomics Hi-C kit (see the "Material and methods" section). instaGRAAL was run for 15 cycles, with the parameter --levels sets to 5, and the scaffolds were subsequently corrected with instaGRAALpolish. We obtained a total of 1302 scaffolds, out of which 24 have a length ranging from 18 to 239 Mb. These 24 chromosome-level scaffolds are represented in the contact map in Additional file 1: Fig. S9. These scaffolds have an N50 and an NGA50 of 143 Mb, close to the 145 Mb obtained for the reference genome (Table  2 ; the results from  [33]  using SALSA2 are included).
Benchmarking of the system requirements To quantify the improvements made over the original GRAAL program, we ran both GRAAL and instaGRAAL over the Ectocarpus sp. v1 genome separately and measured the peak memory load, the graphics card memory load taken by the contact maps, and the per-cycle runtime as averaged from 20 cycles. The results are summarized in Table  S5. As expected, the memory load on the graphics card is an order of magnitude smaller for instaGRAAL, while the peak RAM and runtime are several times smaller. The shrinkage of memory requirements is predicted by the use of sparse data structures and the fact that our original dataset for Ectocarpus sp. is relatively lean when compared to the size of the genome. The origin of the accelerated runtime is less clear and could be due to multiple contributions to the program, including the use of sparse data structures but also external contributions (e.
The tail filtering correction step inserts these bins back at the extremities of these contigs in the insta-GRAAL assembly. The combination of a probabilistic algorithm with a deterministic correction step provides robustness to instaGRAAL. First, the MCMC step identifies, with few prior assumptions, a high-likelihood family of genome structures, almost always very close to the correct global scaffolding. The correction step combines this result with prior assumptions made about the initial contig structures generated through robust, established assembly programs, refining the genomic structure within each scaffold. To give the user a fine-grained degree of control over our correction procedures, the implementation into instaGRAAL is split into independent modules that each assume about the initial contig structure necessary to perform the correction: the "reorient" module assumes that the initial contigs do not display inversions, and the "rearrange" module assumes that there are no relocations within contigs.
Benchmarking In order to test our tool against existing programs, we ran two scaffolders available online (SALSA2 and 3D-DNA) on our two genomic datasets. In all instances, insta-GRAAL proved more successful at scaffolding both genomes. However, we have not extensively tested all the combinations of parameters of both programs, and acknowledge the difficulty in designing and implementing Hi-C scaffolding pipelines with extensive dependencies that compound the initial complexity of the task and add yet more configurable options to know in advance. Finding the correct combination of CUDA and Python dependencies to install instaGRAAL on a given machine can be challenging as well. Therefore, our benchmarking attempt should be rather seen as a way to stress the importance of implementing sensible default parameters that readily cover as many use cases as possible for the end user.
Contact map generation Contact maps were generated from reads using the hicstuff pipeline for processing generic 3C data, available at https://github. com/koszullab/hicstuff. The backend uses the bowtie2 (version 2. 2. 5) aligner run in paired-end mode (with the following options: --maxins 5 -very-sensitive-local). Alignments with mapping quality lower than 30 were discarded. The output was in the form of a sparse matrix where each fragment of every chromosome was given a unique identifier and every pair of fragments was given a contact count if it was non-zero. Fragments were then filtered based on their size and total coverage. First, fragments shorter than 50 bp were discarded. Then, fragments whose coverage was less than one standard deviation below the mean of the global coverage distribution were removed from the initial contact map.
Validation with the genetic map The validation procedure with respect to linkage data was implemented as part of instaGRAAL. Briefly, the script considers a set of linkage group where regions are separated by SNP markers and a set of Hi-C scaffolds where regions are bins separated by restriction sites. It then finds best-matching pairs of linkage groups/scaffolds by counting how many of these regions overlap from one set to the other. Then, for each pair, the bins in the Hi-C scaffold are rearranged so that their order is consistent with that of the corresponding linkage group. Such rearrangements are parsimonious and try to alter as little as possible. Since there is not a one-to-one mapping from restriction sites to SNP markers, some regions in the Hi-C scaffolds are not present in the linkage groups, in which case they are left unchanged.
Benchmarking with other assemblers For each genome, the 3D-DNA program was run using the run-assembly-pipeline. sh entry point script with the following options: -i 1000 --polisher-input-size 10000 --splitter-input-size 10000. The Hi-C data was prepared with the Juicer pipeline as recommended by 3D-DNA's documentation. The SALSA2 program was run with the -cutoff=0 option, and misassembly correction with the -clean=yes option. No expected genome size was provided. The program halted after 9 iterations for Ectocarpus sp. and 18 iterations for D. herbacea. Hi-C data was prepared with the Arima pipeline as recommended by SALSA2's documentation. The similarity dot plot between corrected instaGRAAL and SALSA scaffolds was generated with minimap2.
Software tool requirements The instaGRAAL software is written in Python 3 and uses CUDA for the computationally intensive parts. It requires a working installation of CUDA with the pycuda library. CUDA is a proprietary parallel computing framework developed by NVIDIA and requires a NVIDIA graphics card. The scaffolder also requires a number of common scientific Python libraries specified in its documentation. The instaGRAAL website lists computer systems onto which the program was successfully installed and run.
Funding This research was supported by funding to R. K. and S. M. C. from the  European Research Council  under the  Horizon 2020 Program  ( ERC grant  agreements  260822  and  638240 , respectively). This project has also received funding from the  European Union 's  Horizon 2020 research and innovation program  under the  Marie Skłodowska-Curie  grant agreement No  764840.
Availability of data and materials The datasets generated and analyzed in the present work are available in the SRA repository, SRR8550777  [48]. The instaGRAAL software and its documentation are freely available under the GPL-3. 0 license at https://github. com/ koszullab/instaGRAAL  [49]. Assemblies, contact maps, and relevant materials for the reproduction of the main results and figures are available at https://github. com/koszullab/ectocarpus_scripts [50].
A. Network Calculus Background Network Calculus describes data flows by means of cumulative functions, defined as the number of transmitted bits during the time interval [0, t]. Consider a system S receiving input data flow with a Cumulative Arrival Function (CAF), A(t), and putting out the same data flow with a Cumulative Departure Function (CDF), D(t). To compute upper bounds on the worstcase delay and backlog, we need to introduce the maximum arrival curve, which provides an upper bound on the number of events, e. g. , bits or packets, observed during any interval of time. Definition 3. 1: (Arrival Curve)  [18]  A function α is an arrival curve for a data flow with the CAF A, iff: ∀t, s ≥ 0, s ≤ t, A(t) -A(s) ≤ α(t -s) A widely used curve is the leaky bucket curve, which guarantees a maximum burst σ and a maximum rate ρ, i.
Using FMCW radars for snow and ice studies has been well established since the 1970s (e. g. ,  [28, [37] [38] [39] ; reviewed in  [29] ). More recently, multifrequency systems  [40]  and imaging FMCW synthetic aperture radar  [41, 42]  have been shown to be very useful. Recent studies have shown improvements in sensors and retrieval of snow depth measurements  [30, 34, 43] , avalanche studies  [44, 45] , snow stratigraphy based on FMCW echo analysis  [32, 46]  or FMCW-based tomography  [47] , and ice thickness monitoring  [48, 49]. Rodriguez et al. [50]  described a light and compact home-made FMCW, but it is not widely available. The sensor we describe here is the IMST sentire™ FMCW K-band 24-GHz radar module commercialized by IMST (Kamp-Lintfort, Germany)  [51]. The innovative aspect of this paper is to demonstrate the extent to which the IMST FMCW radar we used in this study can be used for snow and ice measurements (it was initially developed for operational automotive applications).
Sensor Description The low cost compact commercial radar module tested in this study is the K-band 24 GHz IMST sentire™ sR-1200 Series FMCW manufactured by the IMST company, (Kamp-Lintfort, Germany)  [51]. The main specifications of this radar are summarized in Tables  1  and 2  [51] and the hardware is shown in Figure  2. We only used its FMCW mode for our snow and ice characterization application. A simplified link budget is provided in Appendix B. Figure  1. Schematic illustration of the radar retrieval. On the left, the radar is at a height h air above the snow, that is at a height (h r ) above the ground, given a measured snow depth of h s = 117 cm. On the right, the measured radar signal expressed in "radar distance" is shown, with a series of peaks linked to different layers within the snowpack.
Sensor Description The low cost compact commercial radar module tested in this study is the K-band 24 GHz IMST sentire™ sR-1200 Series FMCW manufactured by the IMST company, (Kamp-Lintfort, Germany)  [51]. The main specifications of this radar are summarized in Tables  1  and 2  [51] and the hardware is shown in Figure  2. We only used its FMCW mode for our snow and ice characterization application. A simplified link budget is provided in Appendix B. The radar technology is based on a monostatic FMCW radar system with transmitter and receiver channels and I/Q demodulator  [26]. The IMST radar module contains two receiving antennas closed together in one module and fed by one source. The output signal thus consists of four values corresponding to the real and imaginary parts of amplitude, the I and Q channels, respectively, for both antennas.
Operating Modes The radar is generally controlled and operated with a field tough book or laptop loaded with the software provided by IMST. However, for several applications involving snow or ice monitoring over remote regions, we developed a light compact system with our own controller. The different components of the system, which allow on board signal processing, are described in Figure  5. 2 1 + -1 (14) The term on the right is an approximation often used when ≪. Note that the penetration depth corresponds to the distance at which microwave power is reduced to 1/e (e = 2. 718) from the ice or snow surface. This is not the maximum detection depth of the sensor. For a 24 GHz radar through pure ice, if we consider the two-way radar signal time = /2, Equation (  11 ) yields = -1. 9298E-02 T + 6.
Operating Modes The radar is generally controlled and operated with a field tough book or laptop loaded with the software provided by IMST. However, for several applications involving snow or ice monitoring over remote regions, we developed a light compact system with our own controller. The different components of the system, which allow on board signal processing, are described in Figure  5. The radar is connected to a home-made printed circuit board (PCB) using Arduino© or to a Raspberry Pi © microcontroller and data logger. It is loaded with processing software allowing the automatic calculation of snow and ice thickness. Data are recorded on the memory board and/or sent wirelessly (via Bluetooth) to a smartphone or a tablet for real-time visualization of radar profiles and the measured snow water equivalent (SWE) or ice thickness.
The system is powered by a small lithium-ion battery and can be switched to a deep sleep state, consuming less than 5 µW between two series of measurements. An audible or visual alarm can be included to warn if the system detects an ice thickness thinner than a critical value. A GPS receiver can be added to include geospatial information in the saved data but is not necessary for calculating SWE or ice thickness. An inclinometer added to the radar unit ensures that it is parallel to the surface target before beginning a series of measurements, and it filters valid data when the radar is moving. The radar is connected to a home-made printed circuit board (PCB) using Arduino © or to a Raspberry Pi © microcontroller and data logger. It is loaded with processing software allowing the automatic calculation of snow and ice thickness.
Data are recorded on the memory board and/or sent wirelessly (via Bluetooth) to a smartphone or a tablet for real-time visualization of radar profiles and the measured snow water equivalent (SWE) or ice thickness. The system is powered by a small lithium-ion battery and can be switched to a deep sleep state, consuming less than 5 µW between two series of measurements. An audible or visual alarm can be included to warn if the system detects an ice thickness thinner than a critical value. A GPS receiver can be added to include geospatial information in the saved data but is not necessary for calculating SWE or ice thickness. An inclinometer added to the radar unit ensures that it is parallel to the surface target before beginning a series of measurements, and it filters valid data when the radar is moving.
Ice Thickness Retrieval from Remotely Piloted Aircraft (RPA) A remotely piloted aircraft (RPA) system can be very useful for monitoring thin ice at the beginning of the winter or over a large lake ice area. The compactness and low weight of the radar studied here allows it to be deployed on off-the-shelf commercial RPAs. A preliminary test was conducted in order to evaluate the performance of measuring ice thickness using the radar onboard a Phantom 2 (Figure  8 ). In this test, the system included a GPS and an inclinometer for recording viewing angle variations (Figure  5 ). The weight of the radar system, including battery, GPS, inclinometer and the control system (Arduino) was 480 g. The data were recorded on a micro-SD card. Weather conditions during the test flight were challenging, with a mean wind speed of 20 km/h (gusting up to 35 km/h) and air temperatures of -15°C.