dataesr/scientific-paragraphs-categorization huggingface dataset:
This work was supported by CS-ORION Marie Curie Industry -Academia Partnerships and Pathways (IAPP) project funded by the European Commission in FP7 (PIAP-GA-2009-251605). It has been co-financed by the European Union and Greek national funds through the National Strategic Reference Framework (NSRF), Research Funding Program: "Cooperation-2011", Project "SeNSE", grant number: 11ΣYN-6-1381.
The method RQS  [15]  exploits a similar idea but, instead of computing the BWT, compute a dictionary of frequently occurring kmers and then identify kmers within small Hamming distance of frequent ones. Positions corresponding to differences from such frequent kmers are assumed to be SNPs or sequencing errors and their quality values are preserved, while the other quality scores are smoothed. Surprisingly, RQS improves SNP-calling accuracy on a gold-standard dataset. However, their method scales to large read sets only if the dictionary is constructed over a sample of the original data set, and the effect of such sampling is not clearly measured. In this paper, we introduce LEON, a novel de novo method for lossless sequence compression and lossy quality compression using methods derived from assembly principles.
However, instead of building a reference as a set of sequences, the reference is represented as a probabilistic de Bruijn Graph. Read order is preserved and each read is represented by a kmer anchor and a list of bifurcation choices, enough to re-build it from the graph. The same data structure is used for quality compression. Nucleotides covered by a sufficient number of highly frequent kmers are assumed to be error-free and have their quality smoothed to an arbitrarily chosen value. The quality stream is then compressed with the zlib library. LEON was run on various real sequencing datasets to evaluate the impact on compression ratio of numerous dataset features, such as the size and complexity of the target genome, the sequencing depth, the sequencing technology. In all cases, LEON showed higher overall compression ratios than state-of-the-art compression software and was at least 2 times faster than the generic tool GZIP, with at least 5 times better compression ratios.
For instance, on a C. elegans whole genome sequencing dataset,the original file size is divided by more than 20, and in a large human dataset case, the file size was reduced from 733 GB to 47 GB. Several types of sequenced samples (whole genome, exome, RNA-seq or metagenomics) were also tested, demonstrating the robustness of LEON. Finally, the effects of the lossy transformation of quality scores was evaluated on a SNP calling analysis and showed an improvement in the prediction accuracy.
Building the reference as a de Bruijn Graph A de Bruijn Graph is a directed graph where each node is a word of length k, called a kmer. An edge is present from node a to node b if the k -1 suffix of node a is exactly the k -1 prefix of node b. A de Bruijn Graph can be built from a set of reads by cutting each read in overlapping kmers. Each read of size l is then a path of lk + 1 nodes in the graph. In this case, the de Bruijn Graph contains as many nodes as there are distinct kmers in the read dataset. Sequencing errors can generate numerous novel distinct kmers that are present in only one or very few reads. This increases drastically the number of nodes in the graph. To avoid this, only kmers that are sufficiently covered in the dataset are represented in the graph, that is kmers having more than T sol (solidity threshold) occurrences in the read dataset, hereafter called solid kmers.
Header compression To compress the sequence headers, a classic compression approach was used. A typical header string can be viewed as several fields of information separated by special characters (any character which is neither a digitn nor alphabetic). Most of these fields are identical for all reads (for instance, the dataset name or the size of the reads). The idea is to store fixed fields only once and efficiently encode variable fields. A short representation of a header can be obtained using its previous header as reference. Each field of the header and its reference are compared one by one. Nothing needs to be kept when fields match. When differences occur, either the numerical difference or the size of the longest common prefix are used to shorten the representation. The resulting short representation is encoded using an order 0 arithmetic coding.
Complexity If we omit the kmer counting step, LEON performs compression and decompression in one single pass over the reads. For a given read, selecting the anchor and building the bifurcation list requires a number of operations that is proportional to the number of kmers in the read. Both compression and decompression processes have running times proportional to the read count multiplied by the average number of kmers per read, that is a time complexity linear with the size of the dataset. It is important to note that decompression is faster than compression. The time consuming kmer counting step is not performed during decompression since the de Bruijn Graph is stored in the compressed file. Two main structures are maintained in main memory during compression and decompression. The bloom filter can use up to G * b bits for storing solid kmers where G is the size of the target genome and b is the number of bits per solid kmers (typically b is set to 12).
Datasets and tools NGS datasets LEON's performance was evaluated on several publicly available read datasets. Main tests and comparisons were performed on whole genome sequencing (WGS) Illumina datasets with high coverage (more than 70x), from three organisms showing a large range of genome sizes and complexities: a bacteria E. coli (G=5 Mbp), a nematode C. elegans (G=100 Mbp) and a human individual (G=3 Gbp). The largest file tested is the WGS human one with 102x coverage resulting in an uncompressed fastq file size of 733 GB. To evaluate the impact of sequencing depth, these datasets were then randomly down-sampled. Additionally, other types of sequencing protocols and technologies were tested, such as RNA-seq, metagenomics, exome sequencing or Ion Torrent technology. Detailed features and accession numbers of each dataset are given in Additional file 1: Table  ST1.
Other tools and evaluation criteria Several compression software were run on these datasets to compare with LEON, from best state-of-the-art tools to the general purpose compressor GZIP (Additional file 1: Table  ST2 ). Tools that are able to compress the whole fastq files and that allow to properly handle paired reads were preferentially chosen. SCALCE and MINCE that re-order reads were run with their option to keep paired reads together. Additionally, being the best read-reordering tool, Orcom was kept for reference. However, it cannot be directly compared to other tools as it loses the read pairing information. Manually concatenating paired reads before compression is possible for ORCOM, however this leads to poor compression ratio. Hence we decided to show ORCOM results only in the mode that lose paired read information.
Lossy quality compression evaluation Lossy quality compression ratios cannot be compared without also taking into account the impact of the lossy transformation of the quality stream on downstream analysis, for instance SNP-calling accuracy. An experiment on a read dataset from the "1000 genomes project" (phase 1 release) was performed. More specifically, SNPs were called before and after several lossy quality transformations on a low coverage Illumina read sequences dataset, corresponding to the human chromosome 20 (HG00096 individual, SRR062634). Five reference-free, lossy quality compression tools were tested together with LEON, each of them being representative for a particular category of methods: FASTQZ that lowers the number of bins of the quality spectrum, LIBC- SAM and FQZCOMP that smooth qualities within a block, and RQS as a tool that, similarly to LEON, uses information extracted from the DNA sequences.
Impact of the parameters and de Bruijn Graph false positives The compression ratio of LEON crucially depends on the quality of the reference that is built de novo from the reads, the probabilistic de Bruijn Graph. In order to evaluate the impact of using an approximate de Bruijn Graph compared to an exact representation, the compression ratio of LEON was computed for several sizes of bloom filters expressed as a number of bits per node. The larger the bloom filter, the fewer the false positives but the more space is needed to store it. Figure  S4  in Additional file 1 shows that the optimal trade-off lies around 10 bits per solid kmer, for the 70x C. elegans dataset. It also demonstrates that correctness of de Bruijn Graph is not essential for compression purposes. The kmer size and the minimal abundance threshold (parameters k and T sol respectively) also impact the compression ratio, as they control the number of nodes and the topology of the exact de Bruijn Graph.
DNA compression ratio with respect to dataset features Figure  4  shows that the compression ratio increases with the sequencing depth. Obviously, the more redundant information is contained in the file, the more LEON can compress it. This is due to the fact that the space occupied by the Bloom filter does not depend on the sequencing depth and is rapidly negligible compared to the initial space occupied by the reads when coverage increases (see also Fig. 3 ). Notably, the compression factor depends also on the sequenced genome size and complexity, with better compression for the small and less complex bacterial Fig. 4  Sequence compression ratios by coverage. Compression ratios obtained by LEON on the sequence stream, with respect to the sequencing coverage of the datasets. The three WGS datasets were down-sampled to obtain lower coverage genome.
In this case the de Bruijn Graph contains more simple paths and bifurcation lists are smaller. Figure  3  shows the relative contributions of each component of the DNA compressed stream for diverse datasets. For WGS datasets, this confirms that the relative contribution of the Bloom filter is low for high coverage datasets, but prohibitive for low coverage datasets (10x). For other types of datasets, the relative contributions vary greatly. For instance, the exome dataset is well compressed since the coverage is very high (more than 1000x) on a very small reference (exons representing around 1 % of the human genome). However, as the capture is noisy and some reads fall outside exons, an important part of the compressed file is taken by un-anchored reads. For the RNA-seq and metagenomic datasets, the bifurcation and un-anchored reads components represent the major part of the compressed DNA stream sizes.
This is due to the heterogeneous sequence abundances in these kinds of datasets. In such cases, sequencing errors cannot be identified solely based on the kmer abundances and the solidity threshold is less effective in simplifying the graph. For instance in the case of RNA-seq, highly transcribed genes are likely to generate parts of the de Bruijn Graph with a high density of branchings, the majority of them corresponding to sequencing errors. Conversely, in the metagenomic dataset, numerous species have a low abundance in the sample and their genome is not represented in the de Bruijn Graph, resulting in a high number of un-anchored reads. Among the tested datasets, three correspond to the same target species (E. coli), sequenced with similar sequencing depths (∼ 115x) but with different sequencing technologies or protocols : Illumina HiSEq 2000, Illumina MiSeq and Ion-Torrent.
In Fig. 3 , one can observe that this factor impacts the DNA compression ratio and the relative contributions of each component. The Ion-Torrent dataset has the lowest compression ratio and this is mainly due to the bifurcation and sequencing errors components. This is explained by the sequencing errors that are mostly insertions and deletions, which are not well handled by the current bifurcation algorithm (an insertion or deletion implies the rest of the read will be encoded as errors), contrary to substitution errors. In the Illumina MiSeq protocol, reads are longer than in the classical HiSeq (250 vs 100 pb). Consequently, for the same amount of DNA, there are fewer reads and therefore fewer anchors to be encoded. This explains the great difference in the relative contribution of the anchor address component.
Note that overall DNA compression ratio are roughly similar between both protocols, but this is due to a higher number of sequencing errors in this particular MiSeq dataset. Since the technologies are evolving to produce longer reads with fewer sequencing errors, this suggests that LEON compression ratio will easily fit the technology evolutions. Lastly, because of the anchor selection procedure, initial read order may theoretically impact compression ratio. However, test showed that LEON compression ratio only varies slightly when changing read order, generally below 1 % variation.
Comparison with other tools For high coverage WGS datasets, LEON obtains the best compression ratio for the whole FASTQ file, i. e. sequence, header and quality streams combined, in comparison to other compression software (see Table  1 ). In particular, with respect to the most used tool, GZIP, LEON compressed file can be up to 7 times smaller than the GZIP one for high coverage datasets. In the large human dataset case, we can save up to 686 GB (the file size drops from 733 GB to 47 GB). Interestingly, although QUIP is similar in approach to LEON, results in terms of sequence compression ratio are much lower than LEON. This can probably be explained by the large amount of reads that could not be mapped to the assembled contigs, either because they were incomplete or too fragmented. As expected, ORCOM, which allows read-reordering, achieves the highest sequence compression ratios.
However, it looses important read pairing information and thus cannot be directly compared to lossless methods. Moreover, it only compresses the DNA sequence part and completely discards header and quality scores. MINCE and SCALCE that both re-order reads but keep read pairing information have a lower compression ratio than LEON on the DNA sequence. It seems than keeping read pairing information without degrading compression ratio is not a simple task for read-reordering methods. To be on par with LEON lossy quality scores compression, other tools were also run in a lossy compression mode when available (see command lines in Additional file 1: Table  ST2 ). LEON achieves much higher compression of quality scores than other tools, 26. 8 on the human dataset, compared to 15x for FASTQZCOMP. Additional comparisons on other types of datasets are shown in Fig.
5. LEON is better than other tools on all datasets, except on the metagenomic one where all tools perform roughly equally bad. In general, the sequence stream takes the most space in the whole compressed file for all tools. Interestingly, this is not the case for the Miseq and ion-torrent datasets. They suffer from higher sequencing error rates, which impact more the quality smoothing than the sequence compression. Concerning running times, DSRC is by far the fastest method. It achieves compression ratios generally lower than other methods, but still up to two times better than GZIP. It is a good choice when running time is the major concern. Apart from DSRC, LEON compression time is about the same order of magnitude as other methods compressing the whole FASTQ file, and a bit faster for decompression.
Discussion In this article, we introduced a new method for referencefree NGS data compression. Whereas the QUIP approach is building a de novo reference with traditional assembly methods, we use a de Bruijn Graph as a de novo reference. This allows skipping the computationally intensive and tricky assembly step, and also allows to map more reads on the graph than would be possible on a set of de novo built contigs. Our approach also yields better compression ratios than context model based methods such as FASTQZ or FQZCOMP, which, in a way, also learn the underlying genome from the context. This can be explained by the larger word size used by LEON. Thanks to the probabilistic de Bruijn Graph, our method is able to work with large kmers, whereas context models are limited to order-14 models due to memory constraints.
The development of an API in the GATB library to read the LEON format on-the-fly without full decompression on disk is under development and will facilitate usage by other tools based on GATB (that could use it as a native input format). Moreover, the LEON compressed file contains more information than just the raw list of reads: the included de Bruijn Graph can be directly re-used by other software. For example, the TAKEABREAK and DIS-COSNP software  [29, 30]  detecting polymorphisms from the de Bruijn Graph will be able to take as input a LEON file and save significant time from the graph construction step. In this way, LEON can be seen as more than just a compression tool, as it also pre-processes data for further NGS analysis. Further developments to enhance LEON performance and functionalities are also considered.
First, if reordering reads is acceptable for the user, grouping reads with the same anchor would allow to store the anchor once for many reads and save significant space. However, readreordering strategy is acceptable in our opinion only if read pairing information is preserved, which is not the case of current read-reordering methods. With LEON method, since paired reads are close in the graph, it may be possible to encode paired reads together, by also encoding the path in-between reads. This would allow read reordering without losing read pairing information. Secondly, the detection of insertion and deletion errors could boost substantially the compression ratio of datasets issued from novel sequencing technologies (Ion Torrent or Pacific Bioscience). Moreover, our approach makes it possible to deal with multiple datasets efficiently.
It would be straightforward to store the de Bruijn Graph only once for several datasets sequenced from the same organism for instance, and thus improving compression ratio. Lastly, our approach bears some similarities with error correction methods. When reads are anchored to the graph, some sequencing errors are clearly identified and saved in the file for the decompression. It could be combined with more powerful error detection algorithms to provide state-of-the art error correction, for example with the BLOOCOO 1 tool already implemented with the GATB library  [23]. It would then be straightforward to propose an option when decompressing the file, to choose between lossless sequence decompression mode, or with the sequencing errors corrected.
Conclusions We introduced LEON, an all-in-one software for FASTQ file compression that handles DNA, header and quality scores. LEON uses the same data structure for both DNA and quality scores compression, a de Bruijn Graph compactly stored in a Bloom filter. The quality compression scheme is lossy, allowing for good overall compression ratios, without losing the essential quality information and thus not hampering downstream NGS analysis. LEON achieves higher compression ratios than other state-of the art tools and seems robust regarding diverse types of data.
Compression features obtained for the three high coverage WGS datasets with several compression tools. Total compression ratio is the compression ratio (original size / compressed size) of the whole FASTQ file, header, sequence and quality combinedThe following columns indicate the ratio for each individual component, when available. Running time (in s) and peak memory (in MB) are given for compression and decompression. All tools were used without a reference genome. Best overall results are in bold a Program does not support variable length sequences b SCALCE was not able to finish on the large WGS human dataset -lossy suffix means the method was run in lossy mode for quality scores compression * Stars indicate that the given program changes read order and loses read-pairing information, and thus cannot be directly compared to other tools Running time with # is on DNA sequence only Best overall results are in bold of the original file (indicated with a dashed line).
Acknowledgements The authors are grateful to  Thomas Derrien ,  Rayan Chikhi  and  Delphine Naquin  for beta-testing. This work was supported by the  ANR-12-EMMA-0019-01   GATB  project and  ANR-12-BS02-0008  Colib'read project. The  GenOuest BioInformatics Platform  provided the computing resources necessary for benchmarking.
†Current address: Centre for Climate Change, Universitat Rovira i Virgili, Tarragona, Spain ‡Current address: Ente Regionale per i Servizi all'Agricoltura e alle Foreste (ERSAF), Milan, Italy §Current address: European Centre for Medium-Range Weather Forecasts (ECMWF), Reading, UK ¶Current address: Department of Space and Climate Physics, University College London, London, UK The International Surface Pressure Databank (ISPD) is the world's largest collection of global surface and sea-level pressure observations. It was developed by extracting observations from established international archives, through international cooperation with data recovery facilitated by the Atmospheric Circulation Reconstructions over the Earth (ACRE) initiative, and directly by contributing universities, organizations, and countries.
The dataset period is currently 1768-2012 and consists of three data components: observations from land stations, marine observing systems, and tropical cyclone best track pressure reports. Version 2 of the ISPD (ISPDv2) was created to be observational input for the Twentieth Century Reanalysis Project (20CR) and contains the quality control and assimilation feedback metadata from the 20CR. Since then, it has been used for various general climate and weather studies, and an updated version 3 (ISPDv3) has been used in the ERA-20C reanalysis in connection with the European Reanalysis of Global Climate Observations project (ERA-CLIM). The focus of this paper is on the ISPDv2 and the inclusion of the 20CR feedback metadata. The Research Data Archive at the National Center for Atmospheric Research provides data collection and access for the ISPDv2, and will provide access to future versions.
Introduction The International Surface Pressure Databank (ISPD) is the world's largest collection of global surface and sea-level pressure observations. Since its inception in 2002, its development has been facilitated by cooperative efforts between the Global Climate Observing System (GCOS)/World Climate Research Program (WCRP) Working Group on Observational Data Sets for  Reanalysis [2007 Reanalysis [ to 2011]] , and the ongoing efforts of the GCOS Working Group on Surface Pressure and the international Atmospheric Circulation Reconstructions over the Earth initiative (ACRE,  Allan et al. , 2011). The full observational record is extracted and assembled from established international archives and newly available collections from more than 60 different contributing organizations, shown in Tables  1  and 2.
The ISPD version 2 (ISPDv2) covers the period 1768-2012. It merges data from three input components: observations from land stations, marine observing systems, and tropical cyclone best track pressure reports. The land station component was extracted from many national and international collections of sea-level pressure and surface pressure. The largest contributor to this component was the Integrated Surface Database  (Smith et al. , 2011) , which consists of global hourly and synoptic surface observations collected from many sources. The land stations were merged following a two-step algorithm to first remove duplicates within a collection and then remove duplicates between collections  (Yin et al. , 2008). The marine component consists of sea-level pressure observations extracted from the International Comprehensive Ocean-Atmosphere Data Set (ICOADS,  Woodruff et al.