dataesr/scientific-paragraphs-categorization huggingface dataset:
A set of self-contained Python packages for metadata handling In an effort to interface the primary metadata systems of RELION  [22]  and Dynamo  [19]  with the scientific Python  [23]  ecosystem, we provide 2 Python packages: starfile and dynamotable. These packages provide input/output functionality via a simple application programming interface (API), exposing data as pandas DataFrame objects. Euler angles are used by many cryo-EM software packages to describe the orientation of a rigid body with respect to a fixed coordinate system. eulerangles is a Python package that provides a simple API for the batch conversion of Euler angles into rotation matrices, rotation matrices into Euler angles, and interconversion of Euler angles defined according to different conventions. Conversions can take place between all possible formulations of Euler angles in a right-handed coordinate system.
Interfacing Dynamo and the Warp-RELION-M pipeline The recommended procedure for particle picking in the Warp-RELION-M pipeline is exhaustive, reference-based template matching. Dynamo offers an interactive environment for picking particles based on supporting geometries in cryo-electron tomograms  [15]. We provide warp2catalogue, a program that sets up a database called a Dynamo catalogue for tomograms reconstructed in Warp. The catalogue is set up such that all visualisation operations make use of a deconvolved reconstruction, filtered for improved visualisation, while subsequent particle extraction uses the corresponding unfiltered volume, simplifying the experience for the user. For rigid body particle pose optimisation, STA in RELION is integrated into the existing pipeline. Alternative approaches to particle picking, pose optimisation, and classification may provide advantages over this workflow.
We provide dynamo2m, a set of tools that create a bidirectional interface between Dynamo and the Warp-RELION-M pipeline. dynamo2warp allows for particle position and orientation data in Dynamo to be used for particle reconstruction in Warp, rigid body refinement in RELION, and multi-particle refinement in M. warp2dynamo provides a route to using particles reconstructed in Warp within Dynamo. An additional utility, relion_star_downgrade, is provided to enable reconstructing particles refined using RELION version 3. 1 or higher in Warp 1. 0. 9.
Tools for simplified metadata handling The ability to test alternative approaches and rapidly iterate is key to optimising complex data analysis workflows like cryo-ET and STA. Exploratory data analysis such as checking the internal consistency of particle positions within a supramolecular assembly after a refinement is often key to understanding the limitations of a given approach. Implementing custom input/ output functionality when attempting to work with a variety of nonstandard metadata presents a barrier to entry for those wishing to interactively explore their data or implement custom analysis routines. The Python packages starfile and dynamotable provide simple interfaces between the metadata systems of RELION, Dynamo, and a wealth of data analysis infrastructure, visualisation tools, and educational resources in the scientific Python ecosystem  [25, 26].
Converting between different Euler angle conventions is often a pain point for those wishing to interface different cryo-EM software packages due to the abundance of ambiguities in their interpretation  [27]. eulerangles simplifies generating rotation matrices from Euler angles for custom analyses and facilitates building interfaces between pieces of software that interpret Euler angles according to differing conventions. Our vectorised implementation is approximately 10× faster than the popular scipy utility scipy. spatial. transform. Rotation, taking 381 ms versus 4. 38 s for the conversion of a million Euler triplets, the eulerangles package thus being more suitable for working with large sets of Euler angles in Python. As an example of the benefits derived from working within this ecosystem, we combine starfile and eulerangles with the packages mrcfile  [28]  (for reading and writing MRC format image files) and napari  [29]  (a fast, powerful multidimensional data visualisation library) to visualise tomograms, particle positions, and their oriented Z-vectors in 3D (Fig  2 ).
A flexible framework for ab initio and geometrical approaches to subtomogram averaging Each STA dataset is different, and analysis presents its own unique challenges. The tools provided in autoalign_dynamo and dynamo2m were designed to simplify cryo-ET data processing where possible and empower researchers with access to a vast array of alternative tooling where it may provide benefit. Combined, these tools enable working within a powerful, flexible framework for more automated, ab initio, and geometrical approaches to multi-particle refinement, linking Warp-RELION-M with Dynamo (Fig  3 ). In this section, we discuss and illustrate the benefits provided by our tools.
Automation of tilt series alignment in Warp-RELION-M with autoalign_dynamo Optimisation of data collection strategies has significantly increased the throughput of cryo-ET data collection in recent years, with datasets now routinely exceeding 100 tilt series. As dataset sizes increase, so does the need for accurate, automated solutions to each and every step of complex data analysis workflows. Accurate tilt series alignment is often achieved using semiautomated procedures that quickly become tedious when faced with a large dataset. Providing the automated, robust fiducial-based tilt series alignment procedures from Dynamo as an on-the-fly solution tightly integrated into the Warp preprocessing pipeline greatly simplifies the generation of accurate 3D reconstructions for downstream data analysis.
Providing access to alternative refinement and classification procedures RELION implements a Bayesian approach to the refinement of one or many 3D volumes from cryo-EM data called maximum a posteriori refinement  [22]. Originally designed for singleparticle cryo-EM data and more recently extended to work with cryo-ET data  [32] , RELION provides solutions for the "refinement" block of the cryo-ET pipeline. The package features an easy-to-use graphical user interface (GUI) and is already in use in a large number of cryo-EM labs. Rather than a data processing pipeline, Dynamo is a collection of powerful tools for working with STA data. With an emphasis on customisation, Dynamo often exposes the user to large numbers of parameters in fully featured GUIs. ByAU : PleaseconfirmthattheeditstothesentenceByinterfacingDynamowiththeWarp À RELION À Mp interfacing Dynamo with the Warp-RELION-M pipeline in the "refinement" block, we provide Warp-RELION-M users access to highly customisable STA and multireference classification procedures, principal component analysis (PCA)-based classification, and myriad data visualisation tools and analysis tools.
Application to the HIV-1 CA-SP1 hexamer (EMPIAR-10164) We illustrate the power of working within this framework on an example dataset by reprocessing a 5 tilt-series subset of EMPIAR-10164 containing immature HIV-1 dMACANC virus-like particles (VLPs). This dataset was contributed to the community by the Briggs group, and this subset has previously been used to benchmark NovaCTF  [17]  and Warp  [10] , resulting in 3. 9 Å and 3. 8 Å reconstructions, respectively. Initial motion correction and initial CTF estimation were performed in Warp with respective spatiotemporal model resolutions of (1, 1, 8) and (2, 2, 1). Tilt series were automatically aligned using autoalign_dynamo before CTF estimation and tomogram reconstruction at 10 Å/px in Warp. VLPs were annotated in Dynamo, and initial estimates of positions and orientations were generated with an interparticle distance of 45 Å, oversampled relative to the approximately 75 Å lattice spacing seen in the tomograms.
A total of 500 particles were extracted and averaged in Dynamo to produce a coarse template. The same data were subject to STA against this template in Dynamo without imposing symmetry during refinement. The resulting average contained a hexagonal lattice. To obtain an initial model, the 6-fold axis of the lattice was aligned to the z-axis, and the resulting volume was symmetrised 6-fold in Dynamo. A total of 28,516 particles were extracted from annotated VLPs and aligned against the initial model for one iteration with a limited angular search range in Dynamo. The resulting particle positions and orientations, visualised in Dynamo, formed regular lattices on the surfaces of each VLP with some less regular areas. Using simple MATLAB scripts, only the 19,810 particles with 3 or more neighbours at the expected interparticle distance were retained for subsequent analysis.
Using dynamo2m, metadata were converted to enable working in Warp-RELION-M. Particle reconstructions were carried out in Warp and 3D refinements in RELION 3. 1, starting with local angular searches of ±15˚. Resolution estimates were measured by FSC between independent reconstructions from random half-sets of the data using relion_postprocess. Particles were (i) reconstructed at 5 Å/px and refined (unmasked) to 10 Å resolution; (ii) reconstructed at 2. 5 Å/px and refined (unmasked) to 5 Å resolution; and (iii) reconstructed at 1. 7 Å/px in Warp and refined (masked to include only the central hexamer) to 3. 8 Å resolution. This intermediate result is indicative of the accuracy of fiducial-based alignment workflows in Dynamo and alignments in RELION. Seven rounds of multi-particle refinement were performed in M at 1.
6 Å/px in a masked region containing only the central hexamer. The first 4 rounds of multi-particle refinement, optimising only for deformation parameters, yielded a 3. 6-Å reconstruction. Three further rounds including optimisation of electron-optical aberration-related parameters and tilt frame alignments refined to a resolution of 3. 4 Å (Fig  5 ). It should be noted that working within this framework readily yielded a 3. 4-Å reconstruction from 19,810 particles in 5 tomograms, without use of an external reference and leveraging the geometry of the system evidenced in the tomograms during alignments. The combination of Dynamo tools and the Warp-RELION-M pipeline, enabled by autoalign_dynamo and dyna-mo2m, rendered this ab initio, geometrical approach a smooth, efficient process. An ab initio approach is not strictly necessary for this dataset, as demonstrated by emClarity  [7]  and Warp-RELION-M  [9].
A comprehensive guide to obtaining a 3. 4-Å reconstruction of the HIV-1 CA-SP1 hexamer Information about best practices and approaches to problems presented by cryo-ET data analysis is constantly evolving and currently divided between the literature, mailing lists, and documentation specific to various pieces of software. We provide, in the form of a rich, living document, a step-by-step walk-through for the procedure used to obtain the 3. 4-Å reconstruction of the HIV-1 CA-SP1 hexamer (https://teamtomo. org/walkthroughs/EMPIAR-10164/ introduction. html). This guide takes a user through downloading a 5 tilt series subset of EMPIAR-10164, preprocessing multiframe micrographs in Warp, aligning tilt series automatically using autoalign_dynamo, generating tomograms in Warp, setting up a Dynamo catalogue for geometrical particle picking using warp2catalogue, picking particles based on supporting geometries in Dynamo, initial model generation in Dynamo, isolating an optimal subset of particles using custom scripts, reconstruction of particles in Warp via dynamo2m, rigid body STA in RELION, and multi-particle refinement in M.
Conclusions The tools and resources presented in this manuscript increase automation within the Warp preprocessing pipeline, simplify metadata handling, and interface two complementary cryo-ET data processing packages. The interface provided by dynamo2m enables working within a powerful, flexible framework for geometrical and ab initio approaches to STA, which can benefit from multi-particle refinement in M. Our tools increase the functionality available to users of both Dynamo and Warp-RELION-M, lowering the barrier to employing more customisable workflows for cryo-ET data processing, which are often key to success. Finally, we have attempted to make structural cryo-ET more accessible by providing a comprehensive, step-bystep walk-through to obtaining state-of-the-art STA results. It is our hope that this online guide can serve as a starting point for newcomers to the growing cryo-ET community, enabling them to more quickly understand and use cryo-ET to solve their specific biological questions.
Acknowledgments We acknowledge  Diamond Light Source  for access and support of the cryo-EM facilities at the  UK's national Electron Bio-imaging Centre (eBIC) , funded by the  Wellcome Trust ,  MRC , and BBRSC. Cryo-ET data acquisition has been supported by iNEXT, grant number  653706  (PID: 2626  to IG), funded by the  EU   Horizon 2020 programme.
This work has received funding from a  European Union 's  Horizon 2020 research and innovation programme  under grant agreement No  647784  to IG. AB is supported by a  University Grenoble Alpes  PhD fellowship and by a  Fondation pour la Recherche Medicale (FRM)  fellowship  FDT202001011069. LG is supported by a PhD fellowship from  Grenoble Alliance for Integrated Structural and Cell Biology (GRAL ,  ANR-10-LABX-49-01 ) funded within the  CBH Graduate School of the University Grenoble Alpes  ( ANR-17-EURE-0003 ). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Methods Model. We chose to build a stochastic agent-based model for two reasons: (1) it would allow us to easily implement nonuniform probabilities over the course of infection and (2) a stochastic model would capture the inherent noise in a system that is presumed to start with a small number of infected cases. We implemented the model in MATLAB using the indicated probabilities and if-then statements. The test was performed with 10,000 individuals to represent the college student body. The model runs daily for 120 days, approximating a semester. The basic structure of the model is outlined in Fig. 2A. Because it is a stochastic model, we perform 100 independent runs (Fig. 2B ), and report the median and 95th percentile results. The model can be found on GitHub at https:// github. com/ Kelley-Lab-Compu tatio nal-Biolo gy/ coron amodel.
Nonuniform false negative rates can delay detection of infected individuals. Why are we concerned about uniform versus nonuniform false negative rate? To illustrate the issue, we can examine the first day of disease progression at which an infected individual is likely to be detected when tested daily. To examine the effect of detectability, we explore a hypothetical population where every person is sick and is tested daily (not the full SEIR model). We compare three different false negative rate dynamics over 14 days of disease progression assuming testing everyone, every day, and we assume symptom onset at day 5. The average false negative rate of each is the same (50. 42%, chosen to match the overall average false negative rate of the Kucirka et. al. data  7  ), but the way the rates change over time differs (Fig.
Background: With the increase in the size of genomic datasets describing variability in populations, extracting relevant information becomes increasingly useful as well as complex. Recently, computational methodologies such as Supervised Machine Learning and specifically Convolutional Neural Networks have been proposed to make inferences on demographic and adaptive processes using genomic data. Even though it was already shown to be powerful and efficient in different fields of investigation, Supervised Machine Learning has still to be explored as to unfold its enormous potential in evolutionary genomics.
Therefore, it might be possible to identify alleged genetic factors related to diseases or to relevant traits by identifying regions in the human genome that are currently under selection  [3]. Moreover, in wild or domesticated species, uncovering the pattern of natural selection in their genomes can help identify the genomic basis of their adaptation, e. g. [5]. Inferring the signature left by natural selection is typically addressed comparing the theoretical expectations predicted in case of random drift (neutral evolution) or natural selection with the real data, and the comparison is based on summary statistics of genetic variability  [6, 7]. New advancements in Big Data Analysis are essential to meet the computational challenge that genomics poses  [8]. Modern Machine Learning methods are able to exploit very large datasets, often represented by images, to find hidden patterns and make accurate predictions.
The purpose of this work is to predict whether a genomic region shows a pattern which is more consistent with a natural selection or a neutral model, assuming that all the other parameters of the evolutionary process are constant. Genomic regions simulated with both models are represented as black and white images by converting the 0,1 matrices of genomic data as described below (Dataset generation section). The rows of these matrices represent the mutations of nucleotide bases of a certain genomic sequence for each individual and the columns represent individuals belonging to a population. In this way it is possible to summarize in a single image information on the allelic frequencies of mutations related to a specific genomic sequence in a population. CNNs are then used to analyze images representing sections of the genome and classify them as produced by a neutral or a natural selection model.
Related work In  [12] , the authors propose a program called ImageGene which identifies positive selection on genetic data using CNNs. Both ImaGene and our system rely on the ms program  [24]  to generate the neutral genetic data. However, to generate selection data we use a modified version of the ms program called mssel made available by  [24]  as explained in the Dataset generation section, while ImaGene uses the msms program  [25]. Both mssel and msms are modified versions of the ms program. Similar to our approach, ImaGene converts genetic data into images and uses a CNN to extract and classify patterns from them. Our approach strongly differs from ImaGene in the way the genetic data are generated and in the way they are converted into images. For example, while we generated binary matrices of the same size including a fixed number of sites regardless of whether they are variant or invariant, the ones generated by ImaGene are of different sizes as they include only variable sites.
ImaGene then transforms all binary matrices into a 128x128 pixel image with considerable loss of information. Unlike  [12] , we converted a single binary matrix into a binary image with the same size which maintains all the relevant patterns useful for its classification. Indeed, we obtain high performance in terms of accuracy using less examples than those used in ImaGene. In  [26]  the authors use a supervised Machine Learning approach, called support vector machine (SVM), to discriminate between genomic regions experiencing purifying selection and those free from a selective constraint using population genomic data. Different from our approach, the authors of  [26]  take as input the Site Frequency Spectrum (SFS) 1  of individuals from the Phase I release of 1000 Genomes Dataset which consist of 14 population samples from diverse global locations.
Dataset generation A dataset was created using the ms program  [24]  which allows the simulation of genetic data based on a demographic model defined by the user without selection so that the trajectory of the allele frequencies is only driven by genetic drift (i. e. , Neutral). Its modified version mssel (made available by  [24] ) allows the effect of natural selection to be added to the demographic model (Selection). Using this program, we simulated DNA sequences of defined length for a sample of individuals taken from a population with a fixed size. In each simulation, the sample of DNA sequences can be linked together by a coalescence tree that goes back to a common ancestor in previous generations  [29]. Along this tree, random mutations may appear that modify a base of the DNA sequence with a probability set by the user.
The final variability pattern of this simulated sample of DNA sequences depends on the population size, the probability of mutation and the demographic history set in the model. In the case of the mssel program, the pattern of 1 Site frequency spectrum (SFS): the distribution of allele frequencies in a population sample. variability of the DNA sequences in the sample will also depend on the intensity of selection, another parameter of the model. In both cases, the program produces arrays of DNA sequences, where one dimension corresponds to the length of the sequence while the other to the number of simulated individuals. Since it is a simulation, it is easy to keep track of the ancestral state and the mutated one for each single base of the DNA sequence and then report the sequences as strings of "0" (ancestral state) and "1" (derived state, i.
Therefore, the resulting images are of size 1000x48 and maintain all the information from the simulated genetic regions. In particular in our case study, using the script available in the experiment repository dataset_creator. py 2  the data for both Selection or Neutral models were generated by iteratively running the following command ms2raster. py -bp lines -s number-matrices -l mode -selstr intensity-of-selection -p path-dataset -i number-individuals, which generates binary matrices as defined previously for training, validation and testing. The parameter bp indicates the length in bases of the DNA sequences to be simulated and corresponds to the number of rows of the generated matrices to be generated. We set bp = 1000. The parmeter -s provides the total number of matrices. The parameter -l indicates the mode, selection, neural.
Experiments Experiments 3  were performed using Python as programming language and Keras (with TensorFlow as the backend) as Framework. Experiments were performed using the K80 NVIDIA GPU. Initially, we considered a network consisting of 2 convolutional layers, each with a pooling layer, and two densely connected layers, of which the first has a fairly high number of units (1024) and the last has a single unit (as we are dealing with a binary classification Several experiments showed that both on small and large datasets the model presented clear signs of overfitting probably due to a lot of weights in model and shallow patterns extracted from the images. This led to performing different experiments on different datasets. The datasets are recapped in Table  1  where 10k means 10,000 and ds means dataset.
Note that an equal number of images were generated for both Selection and Neutral in all datasets. To avoid overfitting while extracting relevants patterns on the images, the structure of the model was reviewed as follows: we expanded the model, as shown in Fig. 7. Further convolutional layers were added (with 32, 64 and 128 filters of dimension  [10, 10]  and stride (2, 2) respectively) to enable relevant patterns extraction. To prevent overfitting, several dropout layers (with rate 0. 4) were also added. The units of the first densely connected layer where also reduced, from 1024 to 128. Initially, Adam was used as the optimizer but led to unsatisfactory results. Then, SGD with momentum = 0. 9 was used as optimizer  [30]. Regarding the hyperparameters, a mini-batch of 100 images were used in the case of the ds1 dataset, while 50 images were used for the remaining datasets.