dataesr/scientific-paragraphs-categorization huggingface dataset:
Indexes can also be updated: compatible indexes (i. e. same k-mer size and Bloom filter size) can be merged into a single index, saving query time. Being able to add novel samples to an existing index is an important feature when dealing with growing sample collections, yet rarely present in current indexing tools. The design of kmindex enables it to be used as a web server, using a command line interface, or via a Python API. Depending on the user request, for each indexed sample, the answer to a query sequence can be either its ratio of k-mers shared or, more precisely, the positions where these shared k-mers occur. In the same spirit, when more than one sequence is queried, the answer for each indexed sample can be either a single average number of shared k-mers, or a value per input queried sequence.
Comparative results indexing 50 metagenomic seawater samples We evaluated the performance of kmindex together with state-of-the-art k-mer indexers MetaGraph  [11] , MetaProFi  [20] , and PAC  [16]. We first indexed raw metagenomic seawater sequencing data from 50 Tara Oceans samples, composed of 1. 4TB of gzipped fastq files. Each sample is represented by several distinct fastq. gz read files (on average 2. 5 files per sample). This dataset contains approximately 1,420 billion k-mers. Among them, approximately 394 billion are distinct, and 132 billion occur twice or more. All used command lines and links to publicly available datasets are provided in a companion GitHub website https://github. com/pierrepeterlongo/kmindex_benchmarks.
kmindex has superior index construction performance Table  1  shows that kmindex is 5-10x faster than other tools for indexing these 50 Tara Oceans samples, and uses 2-3x less memory and disk. There were several caveats relative to the other methods. We report construction results for PAC despite the software suffering from a critical bug and returning empty queries, resulting in a 100% query false negative rate. For MetaProFi, as it cannot filter k-mers we used KMC3 to remove k-mers seen only once and thus considered as erroneous before constructing the MetaProFi index. This step is important as 67% of the k-mers are unique in this dataset. Indexing them would have more than doubled the final index size. As MetaGraph needed more than the available amount of RAM (900GB), it could not finish creating the index.
Query results: time and memory performance We tested the scalability of queries with many short-length sequences. For creating the queries, among the 50 Tara Oceans samples, we selected randomly and uniformly between one to ten million reads that Table  3 : Query time performance of the indexes on the 50 Tara Ocean samples. The queries are composed of random reads uniformly sampled from the 50 Tara Oceans datasets. Executions were performed on a cold cache (full results are proposed in the Supplementary Materials). PAC uses partitioning and only loads into memory the partitions that are relevant for the queries resolution. The query times seem to point out that the k-mer diversity results in the loading of almost all partitions once the number of queries reaches 100. kmindex and MetaProFi rely on memory-mapped files, thus the memory usages do not reflect the minimum amount of memory required for the query but also include page caching.
were then used as query sequences. Table  3  shows that kmindex is able to perform thousands of queries on such a large database in a matter of seconds, with negligible RAM usage. This offers the possibility to provide public servers querying in real-time such large datasets, as presented in the following section. These results also show that kmindex outperforms MetaProFi and PAC, both in terms of memory resources and computation time, again with several orders of magnitude. Recall that PAC only reported incorrect results (no hits). As kmindex can query millions of reads in a matter of four to five minutes, this unlocks the possibility of comparing separately each read from a full read set to all data indexed in large projects, such as Tara Oceans. This opens the doors to novel usages for analyzing raw read sets as queries, such as determining the diversity of a complex queried read set or clustering reads based on their similarity to each indexed dataset.
Indexing 1,393 Tara Oceans samples in the Ocean Read Atlas web server Thanks to these novel possibilities offered by kmindex, we built and made available a public web interface able to perform queries on a dataset composed of 1,393 samples (distinct locations and distinct fraction sizes) of the Tara Oceans project  [21]. These samples are divided into six distinct groups, determined by the size fraction of the sequenced species. Based on this clustering we built six distinct indexes (all with the same parameters). At query time, as all the six indexes are registered in a unique meta-index, the whole set of samples is queried. A description of the dataset is available in Table  4. The size of the final uncompressed index is approximately 13% of the size of the raw fastq. gz files, which is 36.
Fraction No This interface is built upon the so-called kmindex-server module. The user can use one or more sequences to query the index and determine the similarity of each sequence to the 1,393 indexed samples. A world map depicts the resulting biogeography, as well as the environmental parameters associated with the sequences. Finally, results can be downloaded in the form of tables or under several image formats. For reasons of robustness and continuity of service, the index is deployed on a networked and redundant filesystem with lower performance than the benchmark environment. As so, the query times are higher than shown in previous results although suitable for this type of service. Details about indexed read sets, and more information about the server architecture and setup are provided in Supplementary Materials The resulting web server named "Ocean Read Atlas" (ORA), whose representation is provided in Figure  1 , extends the "Ocean Gene Atlas" server (OGA)  [23, 22]  that supports queries to assembled genes from Tara Oceans  [21]  and Malaspina  [1].
The first choice is well adapted when indexing sets of samples with distinct characteristics such as their size of variability. Versatile k-mer filtration Convenient ways to filter k-mers are included in kmindex. This feature is based on kmtricks  [14]. It enables the filtration of erroneous k-mers, not only relying on their abundance in a dataset but also on their co-abundances in all indexed datasets. This enables to "rescue" low-abundance k-mers that would have otherwise been removed. To the best of our knowledge, no other indexing tool can integrate this feature. Arbitrarily large k values There is no upper limit on the value of k used for indexing k-mers. We recommend using a high enough value of k so that query results are specific -typically, larger than the logarithm in base 4 of the total length of the genomes present in the collection.
High accessibility The kmindex tool is easy to install via Conda, docker, nix, or portable binary releases in addition to the installation from sources. The kmindex repository is highly documented. Once indexes are built from the command-line interface (CLI), queries can be performed also via a CLI, via an API, or as an HTTP server. This gives users multiple options for interacting with the software, making it easier to integrate into their workflows. The results presented in this manuscript involve metagenomic datasets, however kmindex is technically adapted for indexing any kind of dataset represented using the nucleic alphabet, including barcodes and metatranscriptomics.
Conclusion We propose kmindex, a tool for creating k-mer indexes from terabyte-sized raw sequencing datasets. It is the only tool able to index highly complex data such as thousands of seawater metagenomic samples, and it is the only tool able to provide instant query answers, with a non-zero but negligible false positive rate, in average below 0. 01% in our tests. We believe that by its performance and its usage simplicity, kmindex makes indexing k-mers from large and complex genomic projects practically possible for the first time. Future work includes the compression of the provided indexes, the ability to store each indexed k-mer with its abundance in each dataset, as well as with third-party pieces of information such as known variants, or annotations. Figure  1 : Capitalizing on the kmindex features, we propose "Ocean Read Atlas", a web server able to query one or several sequences provided by the user against the whole Tara Oceans metagenomic dataset in a matter of few seconds, and online.
Overview Conceptually, the presence of each indexed k-mer is stored in one Bloom filter per input read set. At query time, the existence of each k-mer of the query is provided by the Bloom filters. The k-mer counting and filtering processes used for creating the Bloom filters are adapted from kmtricks modules  [14]  and are not recalled in this manuscript. The main kmindex conceptual novelties are how Bloom filters are represented and organized in memory, and how queries are performed, as described in the next sections. Importantly, k-mers are processed according to their sorted hash values during indexing and querying processes. As so, cache misses are significantly minimized when feeding or reading a Bloom filter, hence loading the entire indexed collection in memory is not needed. This results in orders of magnitude faster indexing, and query, and lower memory usage than other k-mer indexing tools.
Index construction The index construction is entirely delegated to kmtricks, which allows partitioned construction of onehash Bloom filter matrices. Each sub-matrix indexes a subset of k-mers matching a specific set of minimizers. As represented in Figure  2  (right), the resulting index consists of P distinct matrices (with P being the number of partitions, equal to 3 in the figure). In order to save indexing and query time, the index is "inverted": given a k-mer, the N bits indicating its presence/absence in the N indexed datasets are consecutive in the index. This allows for fast queries across numerous datasets. Hence, in practice, in a matrix, each row is a bit vector representing the presence or absence of a hash value in each indexed sample. Note that the rows are not packed to save construction and query time.
This results in the fact that each row is composed of N 8 × 8 bits. Doing so, min(0, 8 -N mod 8) bits are unused for each row, as represented by a double arrow in Figure  2. This is up to 7 bits per row. These few lost bits may appear as a drawback, but this is negligible in regard to the N value that is meant to be in the order of a few hundred or thousands, and, importantly, this enables us to efficiently append novel indexed samples to an existing index. By default, the resulting index is not compressed. Although requiring more space, this ensures optimal access time (both for writing and for reading), and it offers the possibility to dynamically append new datasets to an existing index.
Acknowledgements We acknowledge the GenOuest bioinformatics core facility (https://www. genouest. org) and the TGCC (https://www-hpc. cea. fr/index-en. html) for providing the computing infrastructure, as well as  France Génomique  for funding of the TGCC computing resources used to process data used in this article. The authors thank  Jean-Marc Aury  for his help regarding the usage of the Tara Oceans data sets. Tara Oceans (which includes both the Tara Oceans and Tara Oceans  Polar Circle  expeditions) would not exist without the leadership of the  Tara Ocean Foundation  and the continuous support of Tara Oceans consortium members. The authors also thank  Kahles Andre  and  Mustafa Harun  for their help regarding the usage of MetaGraph, and  Camille Marchet  and  Antoine Limasset  for their support using PAC.
Funding The work was funded by  ANR SeqDigger  ( ANR-19-CE45-0008 ), the  IPL Inria Neuromarkers , and received some support from the  French government  under the  France 2030 investment plan , as part of the  Initiative d'Excellence d'Aix -Marseille Université -A * MIDEX -Institute of Ocean Sciences  ( AMX-19-IET-016 ). Inria has taken up the publication charges for this paper in the context of its open science policy. This work is part of the  ALPACA  project that has received funding from the  European Union 's  Horizon 2020 research and innovation program  under the  Marie Sk lodowska-Curie  grants agreements No  956229  and  872539  (PANGAIA). R. C. was supported by  ANR Full-RNA, Inception  and  PRAIRIE grants  ( ANR-22-CE45-0007 ,  PIA/ANR16-CONV-0005 ,  ANR-19-P3IA-0001 ).
MDTOMO allows obtaining an atomic-scale model of conformational variability and the corresponding free-energy landscape, from a given set of cryo-ET subtomograms. The article presents the performance of MDTOMO on a synthetic ABC exporter dataset and an in situ SARS-CoV-2 spike dataset. MDTOMO allows analyzing dynamic properties of molecular complexes to understand their biological functions, which could also be useful for structure-based drug discovery. Cryo electron tomography (cryo-ET) is a structural biology technique that allows determining the three-dimensional (3D) structure of abundant biological macromolecular complexes. This technique is particularly important as it allows studying the complexes and their interactions in situ, i. e. , in their native cellular environment 1. In cryo-ET, a series of images are recorded by titling the sample over a range of angles.
The Gō-like models are a class of models that share similar concepts as the model proposed by Gō (e. g. , defining the long-range interactions based on the native contacts of the initial model, which tends to bias the simulation towards the reference). Cα Gō model represents the protein with Cα atoms of the amino acid chain and captures better the native dynamics than the original Gō model  27. Here, the coarse-grained models were obtained by the SMOG2 software  28.
Low-dimensional conformational space and free-energy calculation. As MDTOMO results in one atomic model per subtomogram, a further analysis must be performed to visualize the conformational landscape. In this context, the set of atomic models obtained by MDTOMO can be projected onto a low-dimensional (e. g. , 2D or 3D) space. In this study, two methods were used to perform the dimension reduction of the atomic coordinates, namely Principal Component Analysis (PCA) and Uniform Manifold Approximation and Projection (UMAP)  29. PCA is the well-established technique for dimension reduction, offering a fast, deterministic and linear reduction method, which is useful to describe the overall variability in most datasets. UMAP is a recent method that allows capturing the non-linear motions, which can, in some cases, achieve a better separation of the different populations compared to PCA.
Concerning the in situ dataset of SARS-CoV-2 spikes, we used the STA alignment that had been obtained in the original work  22. The rigid-body pre-alignment is refined in MDTOMO by estimating the rigid-body rearrangements that occurred during the MD simulation, and using them to correct the initial alignment. The estimation of the rigid-body alignment parameters is obtained by optimizing the rotation and the translations that minimize the root mean square deviation (RMSD) between the initial conformation and the fitted conformation, using an optimization algorithm based on singular value decomposition that is available in BioPython  34. Software implementation. MDTOMO is implemented as part of ContinuousFlex  35  , a software package for continuous conformational heterogeneity analysis in SPA and cryo-ET data, which is also available as a plugin of Scipion  33.
Results In this section, we show the performance of MDTOMO using a set of synthetic subtomograms of an ABC exporter  21  and using a set of experimental in situ subtomograms of a SARS-CoV-2 spike protein (S)  22. Experiment with a synthetic dataset of ABC exporter. Synthetic dataset of ABC exporter. To test the performance of MDTOMO, we synthesized a dataset as close as possible to experimental cryo-ET conditions. The dataset simulates a highly heterogeneous cryo-ET dataset of a heterodimeric ABC exporter, TmrAB, along the substrate translocation cycle. During the translocation cycle, the ABC exporter changes its conformation from outward-facing to inward-facing, allowing the substrate to enter the intracellular cavity. ATP-binding first induces closing of the intracellular gate (an occluded conformation of the exporter, where both intracellular and extracellular gates are closed) and, then, opening of the extracellular gate (an outward-facing conformation, where the extracellular gate opens while the intracellular gate remains closed), which allows a release of the substrate in the extracellular environment.
The structure of TmrAB in inward-facing, outward-facing, and occluded conformations were derived from cryo-EM maps  21  and are available in the Protein Data Bank (PDB) under the accession codes 6RAF, 6RAH, and 6RAK, respectively. Figure  1a  shows the PDB: 6RAK structure of TmrAB in the occluded conformation while Fig. 1b  shows a sketch of the translocation cycle, with all three mentioned conformational states of TmrAB during the cycle. The conformational heterogeneity was generated by standard MD-based flexible fitting (without normal mode acceleration) to obtain a dataset with realistic conformational variability, composed of a number of more populated conformational states (representing more stable states), close to the three mentioned states of TmrAB during the translocation cycle, and a number of less populated, intermediate states between these more stable states.
Nevertheless, the dataset consisting of a continuum of conformational states visited during this MD simulation is rich and representative of the translocation cycle enough for performing the tests of the method that is proposed here. It should be noted that we make the problem of recovery of the ground-truth motions more challenging by using NMMD without adding normal modes to MD simulation in the case of data synthesis and by adding normal modes to MD simulation in the case of fitting (next subsection). Additionally, the Cα-Gō model allows a large enough number of structural degrees of freedom to avoid a bias of synthetic data and the use of two different methods to simulate density maps from the Cα-Gō model in the data synthesis task and in the fitting task makes the ground-truth motions retrieval even more difficult.
Recovery of the ground-truth conformational space. We applied MDTOMO to each of the synthetic datasets (one for each SNR value) composed of 3000 subtomograms using 100-ps NMMD simulations incorporating the 10 lowest frequency normal modes, with a force constant of 5000 kcal/mol starting from the PDB:6RAK model (occluded conformation). Figure  1c  shows the free-energy landscape along the first two PCA components for the synthetic MD trajectory (ground truth) and the free-energy landscapes obtained with MDTOMO for each of the three datasets (subtomograms obtained from the images with the SNR of 0. 05, 0. 03, and 0. 01). We observe that MDTOMO was able to recover the conformational landscape very accurately for the datasets with the SNR of 0. 05 and 0. 03. Indeed, Fig. 1c  shows that the most dense regions of the conformational space (i.
e. , the regions of the lowest free energy) are those that are the closest to the three given PDB structures (6RAF, 6RAH, and 6RAK) that were used to target the MD simulations during the data synthesis. In the case of the dataset with the highest level of noise (SNR of 0. 01), the outward-facing conformation was detected less accurately (some densities are missing for the outward-facing state) than the two other states (occluded and inward-facing). Recovery of the rigid-body alignments. The final rigid-body alignment, obtained by MDTOMO, was compared with the ground truth rigid-body displacement (the rotations and shifts used to synthesize the data). The initial rigid-body alignment, done by STA, was also compared with the ground truth rigid-body displacement. The initial and final alignment errors are compared in Fig.
1d, e , which show respectively the rotational and translational errors for the three datasets (three SNR values). We observe that, for both rotational and translational parameters and each value of the SNR, MDTOMO allows a refinement of the rigid-body alignment (the final errors are smaller than the initial errors). Experiment with a dataset of SARS-CoV-2 spike protein in situ. In situ dataset of SARS-CoV-2 spike protein. The SARS-CoV-2 spike (S) protein is a trimer of three identical glycoproteins (Fig. 2a ). It is involved in the cell infection as it mediates the viral entry by binding to the angiotensin-converting enzyme 2 (ACE2) receptor on the cell surface  41, 42. The S-protein head includes three receptor binding domains (RBDs) that are located at the top of the spike and exists in different conformations on the viral capsid.
We used MDTOMO to analyze a set of experimental subtomograms of the S protein in situ that had previously been obtained and analyzed using STA and 3D classification (the corresponding tilt-series data are available in the EMPIAR database; accession code: EMPIAR-10453)  22. In that previous publication  22  , the tilt series were reconstructed into tomograms with 3D-CTF correction  43  , 20,830 subtomograms were picked and classified several classes, out of which two classes were further described. They correspond to two distinct conformational states, out of which one has three closed RBDs (fully closed state) and the other has one RBD opened (the classes containing 8273 and 4321 subtomograms, respectively). MDTOMO was used to analyze the entire set of 20,830 subtomograms, which were first downsampled from their original size (256 3 voxels) to the size of 128 3 voxels and then pre-aligned using the rigid-body alignment parameters obtained by STA in the previous publication  22.
Note that we chose to split the trajectory in five states as this is the minimal number of states for obtaining a comprehensive description of the different intermediate states in this dataset. More precisely, the set of 20,830 MDTOMO-derived conformational models was extended three times, by applying n × 120 • rotation around the z-axis with n = {0, 1, 2} , resulting in 62,490 models (20,830 triplets, each triplet composed of the model rotated by n × 120 • ). Each of the three models in a triplet was compared by RMSD with the five models from the reference trajectory resulting in 15 RMSD values (five RMSD values per model of a triplet). The lowest value of the 15 RMSD values was identified and the associated rotation factor n was collected to determine the angle for the realignment of the MDTOMO-derived conformational model.
Discussion In this article, we presented MDTOMO, a novel approach to explore continuous conformational landscapes of biomolecular complexes by analyzing subtomogram datasets. MDTOMO uses efficient, coarse-grained and normal-mode-empowered MD simulations to extract the conformational states from subtomograms, which are typically averaged out during classical STA, and allows reconstructing low-dimensional (usually 2D or 3D) energy profiles of the structures from the experimental subtomogram data. MDTOMO is the first method for analyzing subtomograms using MD-based flexible fitting. MDTOMO starts by extracting an atomic model from each individual subtomogram, by NMMD flexible fitting of a given reference atomic model into the subtomogram. The extracted atomic models are then projected onto a low-dimensional space, which can be considered as an essential conformational space.