dataesr/scientific-paragraphs-categorization huggingface dataset:
Demultiplexing and Sequence Analyses Demultiplexing was done using customized workflows in Geneious V11. 0. 4  (Kearse et al. , 2012). Reads were separated by primer tags with a maximum of one mismatch and a minimum of 2 reads per tag. Primers were trimmed and reads were aligned together with MUSCLE 3. 8. 425 using eight iterations. The two amplicons B_R and C_F were merged together by De Novo Assembly with four maximum ambiguities and two base pairs gap sizes over the 85 bp overlapping region, and the consensus was then saved in separate folders mirroring wells of sample plates for further curation of the sequences. To do so, we blasted each consensus against all barcode records on BOLD and NCBI. Prior morphological identification established by experts in the collection was used to control the blast results to species or to genus level, depending on the availability of DNA barcodes for closely related species.
Genetic Analyses To analyse the genetic distances between our newly generated sequences, we mined data of matching species or genus from existing French, German, and Finnish reference libraries  (Pentinsaari et al. , 2014; Hendrich et al. , 2015; Rougerie et al. , 2015). This joined dataset of 1,920 sequences and 490 species is available at dx. doi. org/10. 5883/DS-COLSAPRO. The taxonomic tree for the total dataset is provided in Supplementary Material 3.
All records (including failed samples) are publicly available in project PSFOR on BOLD, and all sequenced individuals can be found in dataset dx. doi. org/10. 5883/DS-NEWCOLEO and in the Table in Supplementary Material 1. Using the C_LepFol primers targeting the full-length DNA barcode, 170 (32. 63%) samples produced visible PCR products on agarose gels and were sent for sequencing with Sanger technology. Eventually, 115 specimens (22. 1% of the 521 samples) yielded long and high-quality sequences (mean length = 655. 5; sd = 12. 7; <1% N), of which 104 (90%) had also been successfully sequenced using the HTS approach. Overall, the quality of Sanger sequences is higher with only 3 ambiguous bases over 115 sequences (0. 026 N per sequence), while consensus sequences from Illumina MiSeq reads include a total of 61 ambiguities distributed among the 286 DNA barcodes recovered (0.
Reliability and By-Products of Illumina MiSeq DNA Barcodes Multiple contigs were often retrieved from Illumina sequencing. Across all the samples processed, the mean number of contigs per sample after demultiplexing in Geneious was 9, ranging from no sequence recovery to a maximum of 196 contigs for one sample. High discrepancies in sequence number were also observed varying from 1 read per contig to thousands, but generally tended to be in low proportion for by-products compared to the barcode of interest. These different contigs were either lower quality reads clustered apart, chimeras, contaminations, potential heteroplasmy, or bacterial sequences. Our samples being degraded and collected within working collections, where specimens are handled on a regular basis, many co-amplified human DNA or other contaminants from fresh organisms processed in the lab at the same time were found.
Genetic Distance Analyses Overall, we produced 297 new DNA barcodes, of which 180 are full-length (658 bp). From this, 286 were sequenced using our HTS approach, 104 with both HTS and Sanger sequencing, and 11 only with Sanger sequencing after amplification of the full-length DNA barcode fragment. These DNA barcodes represent 199 different species (58% of the species processed), of which 103 are new additions to the reference library for French saproxylic beetles; these new sequences also represent 82 new BINS for BOLD. The genetic analysis of these 297 newly generated barcodes along with the 1,623 sequences mined from BOLD shows that the means of within-species and within-genus distances are 1. 11 and 13. 62%, respectively. Within species, genetic distance ranges from 0 to 18. 70% whereas we observed 0% to >30.
In particular, we did not observe the increase in success that we had expected considering the high sensitivity of Illumina sequencing and the ability to handle co-amplifications when analyzing HTS reads, whereas these jeopardize the use of the electropherograms produced with Sanger. Nonetheless, it is interesting that HTS produces multiple products, even in low abundance, that can permit detection and documentation of potential heteroplasmy, pseudogenes or, as exemplified in our dataset, of Rickettsia and Wolbachia infections  (Shokralla et al. , 2014; Lanner et al. , 2019). Our sequencing depth and read quality did not allow us to have enough information to confirm the occurrence of heteroplasmy in our samples. Nevertheless, these non-targeted co-amplifications allowed us to confirm the presence of endosymbionts in some species (see Table  in  Supplementary Material 4) and suspect potential heteroplasmy.
With respect to cases of low interspecific divergence (Table  4 ), the higher number of reported cases is actually an artifact of discrepancies in the curation of taxon names in databases. For instance, the absence of divergence between the two specimens of Dorcatoma dresdensis from France and the four specimens of Dorcatoma falli from Germany results from misidentification of the later specimens. Indeed, D. falli is a North American species absent in Europe. Careful examination of the available pictures of voucher specimens of the German D. falli, confirmed that they indeed are misidentified individuals of D. dresdensis. The species complex Isomira murina, I. thoracica, I. hypocrita, and I. semiflava (see Supplementary Material 3) is another example where different species apparently share similar or highly similar DNA barcodes.
This generated DNA barcode dataset of well-curated and identified collection specimens will be helpful for fast and reliable taxonomical identification for potential mass-trapping and broad biomonitoring studies using genetic approaches. Saproxylic beetles are of major interest with respect to forest health concerns and the need for identification at species level is of great importance to link functional traits and ecological patterns  (Gossner et al. , 2013). In total, adding these new sequences to the PASSIFOR dataset  (Rougerie et al. , 2015)  (656 barcodes of 410 species), DNA barcodes reference library now covers 22. 4% (598 species out of 2,663 species) of the French fauna of saproxylic beetles  (Bouget et al. , 2019). When considering records available in BOLD from other European countries, only 1,128 species remain to be barcoded.
ACKNOWLEDGMENTS We would like to thank  Elisabeth A. Herniou  for helpful comments on the manuscript. We are thankful to the three reviewers and to the editor  Prof. Rodney L. Honeycutt  for their helpful comments on the manuscript. Laboratory work was carried out at the  Service de Systématique Moléculaire (SSM) , part of the  Service Unit Acquisition et Analyse de Données pour l'Histoire Naturelle (2AD)  ( UMS2700 ) at the  Muséum national d'Histoire naturelle in Paris.
tuberculosis Erdman (GenBank accession no. NC_020559. 1) using CLC Genomics version 7 (https://www. qiagenbioinformatics. com). Annotation using Prokka version 1. 12 https://github. com/tseemann/prokka) yielded 4,171 protein-coding genes, 2 repeat regions, and 55 RNA genes (51 tRNA, 3 rRNA, and 1 tmRNA). Singlenucleotide polymorphism (SNP) comparison with different lineages at the core genome level using TB profiler (http://tbdr. lshtm. ac. uk) indicated that strain P7739 (Gen-Bank accession no. CAABOY010000000) was related to M. tuberculosis lineage 4 and did not encode for antimicrobial resistance-associated mutations. We isolated M. tuberculosis strain P7739 from this patient using the shell-vial assay 9 days after initiation of TB treatment. Liquid chromatography mass spectrometry (LC/MS) (Acquity I-Class Vion-IMS Q-Tof, Waters, https://www.
The Python package fluidfft provides a common Python API for performing Fast Fourier Transforms (FFT) in sequential, in parallel and on GPU with different FFT libraries (FFTW, P3DFFT, PFFT, cuFFT). fluidfft is a comprehensive FFT framework which allows Python users to easily and efficiently perform FFT and the associated tasks, such as as computing linear operators and energy spectra. We describe the architecture of the package composed of C++ and Cython FFT classes, Python "operator" classes and Pythran functions. The package supplies utilities to easily test itself and benchmark the different FFT solutions for a particular case and on a particular machine. We present a performance scaling analysis on three different computing clusters and a microbenchmark showing that fluidfft is an interesting solution to write efficient Python applications using FFT.
A problem is that for one-dimensional FFT, all the data have to be located in the memory of the process that perform the FFT, so a lot of communications between processes are needed for 2D and 3D FFT. There are two strategies to distribute an array in the memory, the 1D (or slab) decomposition and the 2D (or pencil ) decomposition. The 1D decomposition is more efficient when only few processes are used but suffers from an important limitation in terms of number of MPI processes that can be used. Utilizing 2D decomposition overcomes this limitation. Some of the well-known libraries are written in C, C++ and Fortran. The classical FFTW library supports MPI using 1D decomposition and hybrid parallelism using MPI and OpenMP. Other libraries, now implement the 2D decomposition: PFFT  (Pippig 2013) , P3DFFT  (Pekurovsky 2012 ), 2decomp&FFT and so on.
These libraries rely on MPI for the communications between processes, are optimized for supercomputers and scales well to hundreds of thousands of cores. However, since there is no common API, it is not simple to write applications that are able to use these libraries and to compare their performances. As a result, developers are met with a hard decision, which is to choose a library before the code is implemented. Apart from CPU-based parallelism, General Purpose computing on Graphical Processing Units (GPGPU) is also gaining traction in scientific computing. Scalable libraries written for GPGPU such as OpenCL and CUDA have emerged, with their own FFT implementations, namely clFFT and cuFFT respectively. Python can easily link these libraries through compiled extensions. For a Python developer, the following packages leverage this approach to perform FFT: • sequential FFT, using: numpy.
fft and scipy. fftpack which are essentially C and Fortran extensions for FFTPACK library. -pyFFTW which wraps FFTW library and provides interfaces similar to the numpy. fft and scipy. fftpack implementations. mkl fft, which wraps Intel's MKL library and exposes python interfaces to act as drop-in replacements for numpy. fft and scipy. fftpack. • FFT with MPI, using: -mpiFFT4py and mpi4py-fft built on top of pyFFTW and numpy. fft. pfft-python which provides extensions for PFFT library. • FFT with GPGPU, using: -Reikna, a pure python package which depends on PyCUDA and PyOpenCL pytorch-fft: provides C extensions for cuFFT, meant to work with Py-Torch, a tensor library similar to NumPy. Although these Python packages are under active development, they suffer from certain drawbacks: • No effort so far to consolidate all possible FFT libraries, both sequential, MPI and GPGPU based under a single package with similar syntax.
• Quite complicated even for the simplest use case scenarios. To understand how to use them, a novice user has to, at least, read the FFTW documentation. • No benchmarks between libraries and between the Python solutions and solutions based only on a compiled language (as C, C++ or Fortran). • Provides just the FFT and inverse FFT functions, no associated mathematical operators. The Python package fluidfft fills this gap by providing C++ classes and their Python wrapper classes for performing simple and common tasks with different FFT libraries. It has been written to make things easy while being as efficient as possible. It provides: • tests, • documentation and tutorials, • benchmarks, • operators for simple tasks (for example, compute the energy or the gradient of a field). In the present article, we shall start by describing the implementation of fluidfft including its design aspects and the code organization.
Implementation and architecture The two major design goals of fluidfft are: • to support multiple FFT libraries under the same umbrella and expose the interface for both C++ and Python code development. • to keep the design of the interfaces as human-centric and easy to use as possible, without sacrificing performance. Both C++ and Python APIs provided by fluidfft currently support linking with FFTW (with and without MPI and OpenMP support enabled), MKL, PFFT, P3DFFT, cuFFT libraries. The classes in fluidfft offers API for performing double-precision 2 computation with real-to-complex FFT, complex-to-real inverse FFT, and additional helper functions.
Python API Similar to other packages in the FluidDyn project, fluidfft also uses an objectoriented approach, providing FFT classes. This is in contrast with the approach adopted by numpy. fft and scipy. fftpack which provides functions instead, with which the user has to figure out the procedure to design the input values and to use the return values, from the documentation. In fluidfft, the Python API wraps all the functionalities of its C++ counterpart and offers a more richer experience through an accompanying operator class. As a short example, let us try to calculate the gradient of a plane sine-wave using spectral methods, mathematically described as follows: u(x, y) = sin(x + y) ∀x, y ∈ [0, L] û(k x , k y ) = 1 L 2 L 0 L 0 u(x, y) exp(ik x x + ik y y)dxdy ∇u(x, y) = kx ky ikû(k x , k y ) exp(-ik x x -ik y y) where k x , k y represent the wavenumber corresponding to x-and y-directions, and k is the wavenumber vector.
The equivalent pseudo-spectral implementation in fluidfft is as follows: A parallelized version of the code above will work out of the box, simply by replacing the FFT class with an MPI-based FFT class, for instance fft2d. with fftwmpi2d. One can also let fluidfft automatically choose an appropriate FFT class by instantiating the operator class with fft=None or fft="default". Even if one finds the methods in the operator class to be lacking, one can inherit the class and easily create a new method, for instance using the wavenumber arrays, oper. KX and oper. KY. Arguably, a similar implementation with other available packages would require the know-how on how FFT arrays are allocated in the memory, normalized, decomposed in parallel and so on. Moreover, the FFT and the operator classes contain objects describing the shapes of the real and complex arrays and how the data is shared between processes.
Code organization The internal architecture of fluidfft can be visualized as layers. Through Fig. 2  for the 2D and 3D cases) using the template library mako. Thereafter, Cython  (Behnel et al. 2011 ) generates C++ code with necessary Python bindings, which are then built in the form of extensions or dynamic libraries importable in Python code. All the built extensions are then installed as a Python package fluidfft. A helper function fluidfft. import fft class is provided with the package to simply import the FFT class. However, it is more convenient and recommended to use an operator class, as described in the example for Python API. Although the operator classes can function as pure Python code, some of its critical methods can be compiled, if Pythran (Guelton 2018) is available during installation of fluidfft.
We will show towards the end of this section that by using Pythran, we reach the performance of the equivalent Fortran code. To summarize, fluidfft consists of the following layers: • One C++ class per method derived from a hierarchy of C++ classes as shown in Fig. 1. • Cython wrappers of the C++ classes with their unit test cases. • Python operators classes (2D and 3D) to write code independently of the library used for the computation of the FFT and with some mathematical helper methods. These classes are accompanied by unit test cases. • Pythran functions to speedup critical methods in the Python operators classes. Command-line utilities (fluidfft-bench and fluidfft-bench-analysis) are also provided with the fluidfft installation to run benchmarks and plot the results. In the next subsection, we shall look at some results by making use of these utilities on three computing clusters.
Benchmarks on Occigen Occigen is a GENCI-CINES HPC cluster which uses Intel Xeon CPU E5-2690 v3 (2. 6 GHz) processors with 24 cores per node. The installation was performed using Intel C++ 17. 2 compiler, Python 3. 6. 5, and OpenMPI 2. 0. 2. Fig. 3  demonstrates the strong scaling performance of a cuboid array sized 384 × 1152 × 1152. This case is particularly interesting since for FFT classes implementing 1D domain decomposition (fftw1d and fftwmpi3d), the processes are spread on the first index for the physical input array. This restriction is as a result of some FFTW library internals and design choices adopted in fluidfft. This limits fftw1d to 192 cores and fftwmpi3d to 384 cores. The latter can utilize more cores since it is capable of working with empty arrays, while sharing some of the computational load.
The fastest methods for relatively low and high number of processes are fftw1d and p3dfft respectively for the present case. The benchmark is not sufficiently accurate to measure the cost of calling the functions from Python (difference between continuous and dashed lines, i. e. between pure C++ and the as arg Python method) and even the creation of the numpy array (difference between the dashed and the dotted line, i. e. between the as arg and the return Python methods). Fig. 4  demonstrates the strong scaling performance of a cubical array sized 1152 × 1152 × 1152. For this resolution as well, fftw1d is the fastest method when using only few cores and it can not be used for more that 192 cores. The faster library when using more cores is also p3dfft. This also shows that fluidfft can effectively scale for over 10,000 cores with a significant increase in speedup.
This cluster functions using Intel Xeon CPU E5-2650 v3 (2. 3 GHz) with 20 cores per node and fluidfft was installed using a toolchain which comprises of gcc 4. 9. 2, Python 3. 6. 4 and OpenMPI 1. 6. 5 as key software components. In Fig. 7  we observe that the strong scaling for an array shape of 320 × 640 × 640 is not far from the ideal linear trend. The fastest library is fftwmpi3d for this case. As expected from FFT algorithms, there is a slight drop in speedup when the array size is not exactly divisible by the number of processes, i. e. with 12 processes. The speedup declines rapidly when more than one node is employed (above 20 processes). This effect can be attributed to the latency introduced by inter-node communications, a hardware limitation of this cluster (10 Gb/s). We have also analysed the performance of 2D MPI enabled FFT classes on the same machine using an array shaped 2160 × 2160 in Fig.
8. The fastest library is fftwmpi2d. Both libraries display near-linear scaling, except when more than one node is used and the performance tapers off. As a conclusive remark on scalability, a general rule of thumb should be to use 1D domain decomposition when only very few processors are employed. For massive parallelization, 2D decomposition is required to achieve good speedup without being limited by the number of processors at disposal. We have thus shown that overall performance of the libraries implemented in fluidfft are quite good, and there is no noticeable drop in speedup when the Python API is used. This benchmark analysis also shows that the fastest FFT implementation depends on the size of the arrays and on the hardware. Therefore, an application build upon fluidfft can be efficient for different sizes and machines.
The comment above the function definition is a Pythran annotation, which serves as a type-hint for the variables used within the functions -all arguments being Numpy arrays in this case. Pythran needs such annotation to be able to compile this code into efficient machine instructions via a C++ code. Without Pythran the annotation has no effect, and of course, the function defaults to using Python with Numpy to execute. The array notation is well adapted and less verbose to express this simple vector calculus. Since explicit loops with indexing is not required, the computation with Python and Numpy is not extremely slow. Despite this being quite a favourable case for Numpy, the computation with Numpy is not optimized because, internally, it involves many loops (one per arithmetic operator) and creation of temporary arrays.
In the top axis of Fig. 9 , we compare the elapsed times for different implementations of this function. For this outplace version, we used three different codes: 1. a Fortran code (not shown 10  ) written with three nested explicit loops (one per dimension). Note that as in the Python version we also allocate the memory where the result is stored. 3. a Python version with three nested explicit loops: # pythran export proj_outplace_loop( # complex128[][][], complex128[][][], complex128[][][], # float64[][][], float64[][][], float64[][][], float64[][][]) def proj_outplace_loop  (vx, vy, vz, kx, ky, kz, inv_k_square_nozero) + kz[i0, i1, i2] * vz[i0, i1, i2] ) * inv_k_square_nozero[i0, i1, i2] rx[i0, i1, i2] = vx[i0, i1, i2] -kx[i0, i1, i2] * tmp ry[i0, i1, i2] = vz[i0, i1, i2] -kx[i0, i1, i2] * tmp rz[i0, i1, i2] = vy[i0, i1, i2] -kx[i0, i1, i2] * tmp return rx, ry, rz For the version without explicit loops, we present the elapsed time for two cases: (i) simply using Python (yellow bar) and (ii) using the Pythranized function (first cyan bar).
For the Python version with explicit loops, we only present the results for (i) the Pythranized function (second cyan bar) and (ii) the result of Numba (blue bar). We do not show the result for Numba for the code without explicit loops because it is slower than Numpy. We have also omitted the result for Numpy for the code with explicit loops because it is very inefficient. The timing is performed upon tuning the computer using the package perf. We see that Numpy is approximately three time slower than the Fortran implementation (which as already mentioned contains the memory allocation). Just using Pythran without changing the code (first cyan bar), we save nearly 50% of the execution time but we are still significantly slower than the Fortran implementation. We reach the Fortran performance (even slightly faster) only by using Pythran with the code with explicit loops.
With this code, Numba is nearly as fast (but still slower) without requiring any type annotation. Note that the exact performance differences depend on the hardware, the software versions 11  , the compilers and the compilation options. We use gfortran -O3 -march=native for Fortran and clang++ -O3 -march=native for Pythran 12. Since allocating memory is expensive and we do not need the non-projected velocity field after the call of the function, an evident optimization is to put the output in the input arrays. Such an "in-place" version can be written with Numpy as: # pythran export proj_inplace( # complex128[][][], complex128[][][], complex128[][][], # float64[][][], float64[][][], float64[][][], float64[][][]) def proj_inplace  (vx, vy, vz, kx, ky, kz, inv_k_square_nozero)  As in the first version, we have included the Pythran annotation.