DOI: 10.15482/usda.adc/28083194.v1
title: <i>Salmonella enterica </i>pangenome graph and variant call data for 539,283 genomes

description: <i>Salmonella</i> pangenome graph and variant call data for 539,283 genomesDescription:<i>Salmonella enterica</i> causes human disease and decreases agricultural production. The overall goals of this project is to generate a large database of <i>S. enterica</i> variants with 539,283 samples and 236,069 features for applications in machine learning and genomics. We transformed single nucleotide polymorphism (SNP) data into reduced dimensional representations which are tolerant of missing data based on disentangled variational autoencoders. TFRecord files were made with custom Python scripts that parsed the variant call formats (VCF) into sparse tensors and combined them with the <i>Salmonella</i> In Silico Typing Resource (SISTR) serotype data.The data directory contains:The tar file of TFRecords: <code>tfrecords.tar</code> (103 GB). The TFRecords are organized first by how they were genotyped. <code>mpileup</code> records were created with Mpileup, and the <code>gvg</code> records were created with graph variant calling. In each of these directories batches of ~10,000 sequence reads named Sra10k_XX.tfrecord.gz (00--54). File Sra10k_99.tfrecord.gz contains incomplete SRAs. Each TFRecord contains the shape of the tensor, the indices of non-zero variants, sample name, serotype, and sparse values. Value 99 was assigned to '.' records.The file <code>output.tar</code> (11.4 TB) contains the <code>.vcf</code> files used to create the TFRecords above. The data in here is contained more succinctly in the TTFrecord format. This data will not normally be used.A tar file of metadata files for the samples, <code>metadata</code> (95 MB). Sequence read archive (SRA) accessions were downloaded using <code>edirect/eutilities</code> and saved as <code>SraAccList.txt</code>.<pre><pre>esearch -db sra -query "txid28901[Organism:exp] AND (cluster_public[prop] AND 'biomol dna'[Properties] AND 'library layout paired'[Properties] AND 'platform illumina'[Properties] AND 'strategy wgs'[Properties] OR 'strategy wga'[Properties] OR 'strategy wcs'[Properties] OR 'strategy clone'[Properties] OR 'strategy finishing'[Properties] OR 'strategy validation'[Properties])" | efetch -format runinfo -mode xml | xtract -pattern Row -element Run &gt; SraAccList.txt<br></pre></pre><code>Google BigQuery</code> was used to download metadata for the SRA accessions from the National Institute of Health (NIH).<pre><pre>SELECT * FROM `nih-sra-datastore.sra.metadata` as metadata INNER JOIN `{table_id}` as leiacc ON metadata.acc = leiacc.accID;<br></pre></pre>Files were processed into batches of ~10,000 and named <code>Sra_completed_XX.csv</code> (00--53).A VCF document mapping the TFRecord data to the positions in the graph subjected to the Type strain LT2: <code>mapping/DRR452337.gvg.vcf-with_TFRecord_in_1st_column.txt</code>Scripts for creating and reading TFRecord data: <code>code</code>.<code>reading_and_parsing_fns.py</code> defines functions for converting VCFs of variants called using gvg to sparse tensors and makes the TFRecord files.<code>gvg_to_tfrecord.py</code> creates TFRecords from from the sparse tensors.Tutorial for using the TFRecords: <code>Example_logistic_regression.md</code>Pangenome graph files and references used for variant calling and genotyping: <code>pangenome</code>.<code>refPlus100.fasta.gz</code> which contains the genomes of the 101 Salmonella strains <i>without plasmids</i> used for construction of the pangenome graph.<code>salm.100.NC_003197_v2.d2_complete.gfa.gz</code> The complete 101 Salmonella strain pangenome graph in Graphical Fragment Assembly (GFA2) Format 2.0 including alt nodes used for genotyping<code>salm.100.NC_003197_v2.full.gfa.gz</code> the full graph including alt nodes.<code>salm.100.NC_003197_v2.full.vcf.gz</code> A VCF of the file above<code>genotyped.gvg.vcf</code> the genotype calls in vcf format<code>paths.txt</code> the paths of the graph