dataesr/scientific-paragraphs-categorization huggingface dataset:
Results and Discussion 2. 1. Population Structure, Sequencing and Analysis of Gene Content E. coli were isolated from animals presenting with diarrhoea or mastitis symptoms and subjected to whole-genome sequencing. The original collection consisted of 150 strains that were isolated from 2004 to 2010 and the raw reads from this collection were submitted to the European Nucleotide Archive (ENA) under accession number PRJEB32666. One hundred and forty-seven strains were confirmed as E. coli and 143 strains were included in this study, while four assemblies were discharged due to their poor quality (CFS3241, CFS3306, CFS3348, and CFS3367). The strains represented seven diverse phylogroups (A, B1, B2, C, D, E, and G) as determined by ClermontTyping in EnteroBase  [19]  (Figure  1a ). Bovine mastitis isolates were broadly distributed through all phylogroups, supporting evidence that there are no clearly related genotypes for a proposed bovine mastitis pathotype (Figure  1b ).
The isolates were analysed for the presence of virulence genes through the use of the virulence factor database (VFDB) (http://www. mgc. ac. cn/VFs/, accessed on 28 March 2020), and queried by ABRicate and an E. coli specific database (Table  S2 ). Three strains were positive for the presence of Shiga-toxin genes and different serotypes that have been previously reported in various food-producing animals  [21, 22]  and humans  [23] : the strain CFS3373 was predicted as serotype O88:H25 (ST58/B1/stx1A/stx1vB), the strain CFS3256 as serotype O69:H11 (ST21/B1/stx1A a /stx1B 1b ) and the strain CFS3265 as serotype O2:H27 (ST10/A/stx2A/stx2B). Two strains (CFS3248 and CFS3294) were predicted as serogroup O121, and O45, respectively, two of the seven (O26, O45, O103, O104, O111, O121 and O145) non-O157 serotypes that are associated with serious illness and major outbreaks  [24]  (Table  1 ).
Four E. coli isolates were either positive for mcr-1 as informed by short reading sequencing data or were resistant to colistin by broth microdilution (data not shown), thus E. coli CFS3246, CFS3273, CFS3292, and CFS3313 were sequenced using the PacBio SMRT platform to provide closed genomes for a more in-depth investigation. In addition to the bacterial chromosomes, thirteen total plasmids contained in these isolates were also resolved and queried for features of interest. These included the presence of antibiotic resistance encoding genes, siderophores, and T4SS. A detailed summary of the relevant features of all thirteen plasmids is provided in Table  2. Plasmid sizes ranged from 3. 5 through to 268 kbp. Several replicon types were identified, including IncHI2 (for plasmids pCFS3273-1 and pCFS3292-1), IncF types (for plasmids pCFS3246-1, pCFS3246-2, pCFS3273-2, pCFS3292-2, and pCFS3313-1), and ColRNAI (for plasmids pCFS3246-3, pCFS3273-3, and pCFS3313-4).
Genetic Characterisation of Plasmids pCFS3313-1 and pCFS3273-2 Containing Siderophore-Encoding Genes and ColV Genes Plasmids pCFS3313-1 and pCFS3273-2 encode the iutADCB-iucA-shiF aerobactin siderophore and manganese/iron transport sitABCD operons (Figure  6 ). Figure  6. Siderophore units identified in plasmids pCFS3313-1 and pCFS3273-2. Highlighted in the figure are four selected BLAST hits-plasmids pCOV18, p14EC007b, pCERC4, and pCS0010A. The first two plasmids show a high level of similarity in the flanking regions while the latter two show similarity across the siderophore containing units along with the flanking regions. Colour scheme: siderophore (yellow), metal transporters (blue), transposons and insertion elements (black), toxins/antitoxins (red), and genes of interest (green). Other genes are represented in orange.
The sitABCD operon in both plasmids is identical with a plasmid from an avian pathogenic E. coli  [59]. Plasmid pCFS3313-1 is closely related to the virulence plasmid pAPEC-1 (NC_011980. 1, Table  S3 ). BLAST searches identified a number of plasmids that also contained siderophore-encoding regions (as in the case of plasmids pCERC4 and pCS0010A, identified previously in Australian adults and Salmonella serovars cultured from broiler chickens, respectively) along with other loci (including plasmids p14EC007b and pCOV18, which were identified in French broilers)  [60]. The linear map of these plasmids highlights the conserved siderophore region amongst all sequences except for plasmid pCOV18, which lacks the siderophore module but shows conservation of the flanking virulence gene-encoding regions (Figure  6 ).
In addition to the siderophore modules, these plasmids also shared the genes involved in maintenance and stability regions. However, the flanking regions appear to be largely conserved between the two plasmids. Besides the sporulation initiation inhibitor protein Soj present and conserved between the plasmid pCOV18 and others, there was also conservation in the CcdB-CcdA toxin-antitoxin system shared between plasmids pCOV18 and pCFS3273-2. Figure  6. Siderophore units identified in plasmids pCFS3313-1 and pCFS3273-2. Highlighted in the figure are four selected BLAST hits-plasmids pCOV18, p14EC007b, pCERC4, and pCS0010A. The first two plasmids show a high level of similarity in the flanking regions while the latter two show similarity across the siderophore containing units along with the flanking regions.
Colour scheme: siderophore (yellow), metal transporters (blue), transposons and insertion elements (black), toxins/antitoxins (red), and genes of interest (green). Other genes are represented in orange. The sitABCD operon in both plasmids is identical with a plasmid from an avian pathogenic E. coli  [59]. Plasmid pCFS3313-1 is closely related to the virulence plasmid pAPEC-1 (NC_011980. 1, Table  S3 ). BLAST searches identified a number of plasmids that also contained siderophore-encoding regions (as in the case of plasmids pCERC4 and pCS0010A, identified previously in Australian adults and Salmonella serovars cultured from broiler chickens, respectively) along with other loci (including plasmids p14EC007b and pCOV18, which were identified in French broilers)  [60]. The linear map of these plasmids highlights the conserved siderophore region amongst all sequences except for plasmid pCOV18, which lacks the siderophore module but shows conservation of the flanking virulence gene-encoding regions (Figure  6 ).
Bioinformatics Analysis Raw read quality was assessed with FastQC (version 0. 11. 7) (http://www. bioinformatics. babraham. ac. uk/projects/fastqc/, accessed on 28 March 2020) and low-quality sequences were trimmed using Trimmomatic (version 0. 39)  [71]. Trimmed, paired reads were denovo assembled using SPAdes (version 3. 13. 0) [72]  and the resulting contigs were qualityassessed with QUAST (version 5. 0. 2)  [73]. Analysis of the PacBio data was implemented using SMRT Analysis 2. 3. 0. The best de novo assembly was empirically determined and manually finished by the Hierarchical Genome Assembly Process (HGAP 3. 0) and Canu  [74]. Genomes were checked manually for even sequencing coverage. Afterwards the interim consensus sequence was used to determine the final consensus and accuracy scores using the Quiver consensus algorithm  (16).
Additional metadata and sample IDs are available in Table  S1. Abricate v1. 0. 0 (https://github. com/tseemann/abricate, accessed on 28 March 2020) was used to query genome assemblies with the ResFinder database and VFDB  [79, 80]. The MEGARes  [81] , PlasmidFinder  [82] , and the E. coli specific virulence databases were also queried by Abricate. The combined results into a gene presence/absence matrix and the correspondent values indicating nucleotide percent identity are presented in Table  S2. The plasmid database, PLSDB v. 2020_03_04, was used in a sequence search for related plasmids within the PacBio sequenced genome datasets  [83]  (Table  S3 ). Mash distance (v. 2. 1. 1 on PLSDB, default parameters at maximum p-value = 0. 1 and maximum distance = 0. 1) was used to identify closely related plasmids  [84]  and EasyFig v2.
Introduction Availability of genetic data has increased tremendously due to advances in sequencing technologies and reduced costs  (1). The vast amount of human genetic data is used in a wide range of fields, from medicine to evolution. Despite the advances, cost is still a limiting factor and more data is always welcome, especially in population genetics and genome-wide association studies (GWAS) which usually require substantial amounts of samples. Partially related to the costs but also to the research bias toward studying populations of European ancestry, many autochthonous populations are under-represented in genetic databases, diminishing the extent of the resolution in many studies  (2) (3) (4) (5). Additionally, the majority of the data held by government institutions and private companies is considered sensitive and not easily accessible due to privacy issues, exhibiting yet another barrier for scientific work.
RBMs are two-layer neural networks consisting of an input (visible) layer and a hidden layer (Supplementary Figure  2 ). The learning procedure for the RBM consists in maximizing the likelihood function over the visible variables of the model. This procedure is done by adjusting the weights such that the correlations between the visible and hidden variables on both the dataset and sampled configurations from the RBM converge. Then RBM models recreate data in an unsupervised manner through many forward and backward passes between these two layers (Gibbs sampling), corresponding to sampling from the learned distribution. The output of the hidden layer goes through an activation function, which in return becomes the input for the hidden layer. Although mostly overshadowed by recently introduced approaches such as GANs or Variational Autoencoders  (20) , RBMs have been used effectively for different tasks (such as collaborative filtering for recommender systems, image or document classification) and are the main components of deep belief networks  (21) (22) (23).
Here we propose and compare a prototype GAN model along with an RBM model to create Artificial Genomes (AGs) which can mimic real genomes and capture population structure along with other characteristics of real genomes. These prototypes are compared to alternative generative models based on multiple summaries of the data and we explore whether a meaningful encoding of real data has been learned. Finally, we investigate the potential of using AGs as proxies for private datasets that are not accessible in order to address various genomic tasks such as imputation or selection scans.
Reconstructing genome wide population structure: Initially we created AGs with GAN, RBM, and two simple generative models for comparison: a Bernoulli and a Markov chain model (see Materials & Methods) using 2504 individuals (5008 haplotypes) from 1000 Genomes data  (24) , spanning 805 SNPs from all chromosomes which reflect a high proportion of the population structure present in the whole dataset  (25). Both GAN and RBM models capture a good portion of the population structure present in 1000 Genomes data while the other two models could only produce instances centered around 0 on principal component analysis (PCA) space (Figure  1 ). All major modes, corresponding to African, European and Asian genomes, are well represented in AGs produced by GAN and RBM models and absent for the Markovian and Bernouilli models.
Wasserstein distances between the 2D PCA representations of real versus generated individuals were closer to 0 for GAN (0. 006), RBM (0. 006) and the test set (0. 01) than for the Markovian (0. 124) and Bernoulli (0. 240) models. Uniform manifold approximation and projection (UMAP) mapping results (performed on the combined dataset) lead to similar conclusions (Wasserstein 2D distance from real for GAN: 0. 021, RBM: 0. 091, Markovian: 0. 088, Bernoulli: 0. 127) although the RBM distribution is slightly shifted compare to the real one (Supplementary Figure  3 ). We additionally computed the distribution of pairwise differences of haploid genomes within a single dataset or between the real and artificial datasets (Supplementary Figure  4 ). The real, GAN and RBM distributions present similar multimodal patterns reflecting the underlying population structure (in particular the RBM distribution exhibits three modes corresponding to the average distances between European and Asian, European and African, or African and Asian populations.
However, plotting LD as a function of the SNP distance showed that all models capture weaker correlation, with RBM outperforming the GAN model perhaps due to its slightly overfitting characteristic (Figure  2b ). However, correlations between real and generated LD across all pairs were similar for GAN and RBM (for 1000 Genomes and Estonian respectively: 0. 95 and 0. 97 for GAN, 0. 94 and 0. 98 for RBM) and slightly lower than for test sets (0. 99 and 1. 0) (Supplementary Figure  9 ). LD can be seen as a two-point correlation statistic, we also investigated 3-point correlation statistics, that represent the amount of correlation between triplets of SNPs and thus characterize more complex correlation patterns in datasets (Supplementary Figure  10 ). To further determine the haplotypic integrity of AGs, we performed ChromoPainter  (27)  and Haplostrips  (28)  analyses on AGs created using Estonians as the training data.
AATS score measures whether two datasets were generated by the same distribution based on the distances between all data points and their nearest neighbours in each set. When applied to artificial and real datasets, a score between 0. 5 and 1 indicates underfitting, between 0 and 0. 5 overfitting (and likely privacy loss), and exactly 0. 5 indicates that the datasets are indistinguishable. By using an additional real test set, it is also possible to calculate a privacy loss score that is positive in case of information leakage, negative otherwise, and approximately ranges from -0. 5 to 0. 5. Computed on our generated data, both scores support haplotypic pairwise difference results confirming low privacy loss for GAN AGs with a score similar to the one of an independent Estonian test set never used during training (GAN: 0.
027 ; Test: 0. 021) and the overfitting nature of RBM AGs with a high risk of privacy leakage (RBM privacy loss: 0. 327; Supplementary Figure  13 ). Using an alternative sampling scheme for the RBM (see Material and Methods) slightly reduced privacy loss (restrained under 0. 2 for low number of epochs; Supplementary Figure  14 ). A dataset produced by this alternative scheme had only a slight decrease in quality of the summary statistics while the privacy loss was reduced to 0. 1. For this scheme, the correlation between generated and true allele frequencies was 0. 92 (instead of 0. 95 for the previous RBM and 0. 98 for GAN), the correlation for LD values was 0. 97 (RBM:0. 98, GAN:0. 97), the 2D-Wasserstein distance for the PCA representations was 0. 026 (RBM: 0. 006, GAN: 0. 011, RBM sampling initialized randomly: 0.
Selection tests: We additionally performed cross population extended haplotype homozygosity (XP-EHH) and population branch statistic (PBS) on a 3348 SNP region homogenously dispersed over chromosome 15 to assess if AGs can also capture selection signals. Both XP-EHH and PBS results provided high correlation between the scores of real and artificial genomes (Figure  3 ). Similar peaks were observed for real and artificial genome datasets (see Discussion).
Imputation: Since it has been shown in previous studies that imputation scores can be improved using additional population specific reference panels  (30, 31) , as a possible future use case, we tried imputing real Estonian genomes using 1000 Genomes reference panel and additional artificial reference panels with Impute2 software  (32). Both combined RBM AG and combined GAN AG panels outperformed 1000 Genomes panel for the lowest MAF bin (for MAF < 0. 05, 2. 5% and 4. 4% improvement respectively) which had 5926 SNPs out of 9230 total (Figure  4 ). Also mean info metric over all SNPs were intermediate between the regular imputation scheme (1000 Genomes panel only) and the 'perfect' scheme (panel including private Estonian samples). The scores were 1. 3%, 2. 3%, and 6. 9% higher for the combined RBM, GAN and real Estonian panels respectively, compared to the panel with only 1000 Genomes samples.
Data encoding and visualization via RBM model: Furthermore, similarly to tSNE and UMAP, RBMs perform a non-linear dimension reduction of the data and provides a suitable representation of a genomic dataset as a by-product based on the non-linear feature space associated to the hidden layer (Materials & Methods). As Diaz-Papkovich et al  (33) , we found that the RBM representation differs from the linear PCA ones (Supplementary Figure  15 ), although the general structure identified by the two lower rank components is highly similar. Like in a PCA, African, East Asian, and to a lesser extent, European populations stand out on the two first components yet the relative positions differ slightly from PCA to RBM. In particular, the Finnish appear slightly more isolated from the other European populations on the first component of the RBM.
Comparison with alternative generative models: We additionally performed tests to compare AGs with advanced methods used to generate genomes. One such method is the copying model  (34)  implemented in HAPGEN2  (35). Although genomes generated via this approach performed very well in terms of SFS, LD and PCA, there was extensive overfitting and privacy loss and multiple individuals (747 identical haplotypes) were directly copied from the original dataset (Supplementary Figure  17 ). Another commonly used approach to generate genomes is coalescent simulations. Although it is inherently difficult to make a fair comparison since coalescent simulations require additional (demographic) parameters and do not provide the desired one-to-one SNP correspondence (see Discussion), we compared SFS and LD decay of AGs with genomes simulated via a previously inferred demographic model  (36)  using HapMapII genetic map  (37)  implemented in stdpopsim  (38) (39) (40) (41).
Initially, we performed PCA and checked the allele frequency distribution compared to real genomes (Supplementary Figure  18 ). The reasoning behind PCA was to demonstrate that coalescent simulation genomes cannot be combined with real genomes since they exist in different planes. Since we had selected SNPs for 1000 Genomes and Estonian datasets to be overlapping, we removed alleles below 0. 1 frequency from all datasets to eliminate biases and analyzed LD decay and allele frequency spectrum (Supplementary Figure  19 ). For both summary statistics, coalescent simulation genomes performed well. Still, direct comparison of frequencies SNPs by SNPs, LD pairs by pairs, PCA, AATS or distributions of pairwise distances between real and generated data are not feasible for coalescent simulations. Notably, the demographic model we adopted was optimized for another European population (CEU from the.
Discussion In this study, we applied generative models to produce artificial genomes and evaluated their characteristics. To the best of our knowledge, this is the first application of GAN and RBM models in the population genetics context, displaying an overall promising applicability. We showed that population structure and frequency-based features of real populations can successfully be preserved in AGs created using GAN and RBM models. Furthermore, both models can be applied to sparse or dense SNP data given a large enough number of training individuals. Our different trials showed that the minimum required number of individuals for training is highly variable (i. e. to avoid training failures such as mode collapse or incomplete training without converging to an ideal mode) depending on the unknown dimension of the dataset, which is linked to the type of data to be generated and the population(s).
Since haplotype data is more informative, we created haplotypes for the analyses but we also demonstrated that the GAN model can be applied to genotype data too, by simply combining two haplotypes if the training data is not phased (see Materials & Methods). In addition, we showed that it might be possible to generate AGs with simple phenotypic traits through genotype data (see Supplementary Table and Supplementary Text). Even though there were only two simple classes, blue and brown eye color phenotypes, generative models can be improved in the future to hold the capability to produce artificial datasets combining AGs with multiple phenotypes One major drawback of the proposed models is that, due to computational limitations, they cannot yet be harnessed to create whole artificial genomes but rather snippets or sequential dense chunks.
A technically different approach would be to adapt convolutional GANs for AG generation  (43). Another problem arose due to rare alleles, especially for the GAN model. We showed that nearly half of the alleles become fixed in the GAN AGs in the 10K SNP dataset, whereas RBM AGs capture more of the rare alleles present in real genomes (Supplementary Figure  20 ). A known issue in GAN training is mode collapse  (44) , which occurs when the generator fails to cover the full support of the data distribution. This type of failure could explain the inability of GANs to generate rare alleles. For some applications that depend on rare alleles, GAN models less sensitive to mode dropping may be a promising alternative  (45) (46) (47). An important use case for the AGs in the future might be creating public versions of private genome banks.
Through enhancements in scientific and technology knowledge, genetic data becomes more and more sensitive in terms of privacy. AGs might offer a versatile solution to this delicate issue in the future, protecting the anonymity of real individuals. They can be utilized as input for downstream operations such as forward steps of a specific evolutionary process for which they can become variations of the real datasets (similar to bootstrap), or they can be the sole input when the real dataset is not accessible. Initializing a simulation with real data is a procedure that is commonly used in population genetics  (48, 49). Our results showed that GAN AGs are possibly underfitting while, on the contrary, RBM AGs are overfitting, based on distribution of minimum distance to the closest neighbour (Supplementary Figure  8 ) and AATS scores (Supplementary Figure  13a ), although we showed how overfitting could be restrained by integrating AATS scores within our models as a criterion for early stopping in training (before the networks start overfitting) and by modifying the RBM sampling scheme.
In the context of the privacy issue, GAN AGs have a slight advantage since underfitting and low leakage information is preferable. More distant AGs would hypothetically be harder to be traced back to the original genomes. We also tested the sensitivity of the AATS score and privacy loss (Supplementary Figure  21 ). It appears that both scores are affected very slightly when we add only a few real genomes to the AG dataset from the training set. Although this case is easily detectable by examining the extreme left tail of the pairwise distribution, it advocates for combining multiple privacy loss criteria and developing other sensitive measurement techniques for better assessment of generated AGs. Additionally, even though we did not detect exact copies of real genomes in AG sets created either by RBM or GAN models, it is a very complicated task to determine if the generated samples can be traced back to the originals.