Package: geo-prepper
Submission of high-throughput sequencing data to a public database
such as NCBI GEO (https://www.ncbi.nlm.nih.gov/geo/)
is a critical part of the process of disseminating important scientific
advances to the greater community. With the widespread adoption
of high-throughput sequencing and fast-paced development of associated scientific assays, the scope
and complexity of such projects have increased dramatically, to the point
that prominent scientific articles often employ several different sequencing assays to tell
different parts of the story.NCBI GEO is one of the most widely-used databases of sequencing data. Submissions
will typically include raw data files organized in a particular hierarchy and
a metadata spreadsheet containing details about samples that are part of the
study. The metadata spreadsheet is comprehensive, allowing users to include
arbitrary numbers of samples and metadata. However, filling out this Excel
spreadsheet is tedious and error-prone, as we have to fill in information about
potentially hundreds of samples, with data spread across several different
locations, essentially by hand.To address this problem, here we presentgeo-prepper, a tool to automate
parts of the data submission process to NCBI GEO. Given a samplesheet and config file,
the tool symlinks raw data to a desired output location, and automatically generates
files that can be used to populate the GEO sample submission spreadsheet.Usagepython __init__.py [-h] [-s SAMPLETABLE] [-c CONFIG] [-o OUTPUT_DIR] [-g GROUPING_COL] [-f]

  -s/--sampletable SAMPLETABLE
        Sampletable with sample names, technical replicates (if any), links to raw data and other metadata
  -c/--config CONFIG
        Config yaml file that defines columns of interest in sampletable
  -o/--output_dir OUTPUT_DIR
        Output directory
  -g/--grouping_col GROUPING_COL
        (Optional) Column to group technical replicates. If specified, will override config.yaml specification
  -f/--force
        (Optional) Overwrite output directory if it existsThe tool has three required inputs:config yamlsample tableOUTPUT_DIR: This is the output directory where symlinks to raw data files and other files
are created (seeOutputsection). If the specified directory already exists, the tool
exits with a warning.In addition, there are two optional parameters:GROUPING_COL: This is a column in the sampletable that is used to group technical replicates.
If specified, this overrides theconfig.yamlspecification (seegrouping_colbelow).-f/--force: If specified, this will overwrite an existing output directory.Warning:File paths specified in the sampletable must beabsolute paths.Inputconfig yamlThis is a configuration file with options specified in a yaml format. Here
is a sampleconfig.yamlfor a ChIP-Seq data set:templates/config-chipseq.yamlBelow we list the accepted parameters
of whichsample_col,is_paired_endandfile_colsare required:sample_colThis is used to specify the column containing the sample ID.
The value will be used to match the column names in the
sample table.For example,sample_col: samplenamewill look for the column
namedsamplenamein the sample table for the sample IDs. The
values in this column must be unique.is_paired_endThis is used to specify single-end (SE) or paired-end (PE) data and
must be a boolean (TrueorFalse).file_colsHere we specify the columns containing files to include in the
GEO submission askey:valuepairs. Accepted keys are listed below:keydescriptionR1RequiredRead 1 file (e.g.sample.R1.fastq.gz) for SE or PE RNA-Seq or ChIP-Seq data, or cellranger BAM file (e.g.possorted_genome_bam.bam) for single-cell data.R2Read 2 file (e.g.sample.R2.fastq.gz) for PE RNA-Seq or ChIP-Seq data. Required ifis_paired_endis True.peaksPeak file (e.g.peaks.bed) output by ChIP-Seq peak caller.bigwigBigwig file with ChIP-Seq signalfeaturesFeatures TSV file (e.g.features.tsv.gz) output by cellranger for single-cell data.barcodesBarcodes TSV file (e.g.barcodes.tsv.gz) output by cellranger for single-cell data.matrixMatrix TSV file (e.g.matrix.mtx.gz) output by cellranger for single-cell dataNote I:Of the keys listed above, onlyR1is required. As such, any arbitrary keys can
be added in this section to match specific file types that are part of the submission.Note II:All paths specified in the sampletable must beabsolute paths.metadata_colsA bulleted list specifying additional metadata columns in the sample table. For example,metadata_cols:
    - celltype
    - genotypewill look for the columnscelltypeandgenotypein the sampletable and include them in the outputsample_section.tsv.skip_column_suffixBy default, all keys listed in thefile_colssection, are included as a suffix in the output
(symlinked) file names. So, let’s say, in thefile_colssection, we specified,file_cols:
    R1: 'orig_filename'
    peaks: 'sicer'and in the sampletable, we have the following lines:samplenameorig_filenamesicerwt_1/data/project/seq_core_237_R1.fastq.gz/data/project/peakcaller/peaks_237.bedwt_2/data/project/seq_core_238_R1.fastq.gz/data/project/peakcaller/peaks_238.bedThe output files will be symlinked to the specified output directory (e.g.geo_project) as,geo_project/
  ├─ wt_1_R1.fastq.gz -> /data/project/seq_core_237_R1.fastq.gz
  ├─ wt_2_R1.fastq.gz -> /data/project/seq_core_238_R1.fastq.gz
  ├─ wt_1_peaks.bed -> /data/project/sicer/peaks_237.bed
  └─ wt_2_peaks.bed -> /data/project/sicer/peaks_238.bedSo, thekeyin thefile_colssection (e.g.peaks), is included in the file name as
a suffix (_peaks). To override this behavior, specify the corresponding columns in
theskip_column_suffixsection, as:skip_column_suffix:
    - peaksNow, the peak files don’t have the suffix_peaksin the file name and the
output directory looks like:geo_project/
  ├─ wt_1_R1.fastq.gz -> /data/project/seq_core_237_R1.fastq.gz
  ├─ wt_2_R1.fastq.gz -> /data/project/seq_core_238_R1.fastq.gz
  ├─ wt_1.bed -> /data/project/sicer/peaks_237.bed
  └─ wt_2.bed -> /data/project/sicer/peaks_238.bedgrouping_colThis is used to specify technical replicates (if any). Samples having the same value
in thegrouping_colcolumn, will be considered technical replicates. This is an optional
parameter, and if unspecified, defaults tosample_col.sample tableThis is a TSV containing sample metadata where each row corresponds to an individual sample.
Here is an example sampletable for a ChIP-Seq data set:templates/sampletable-chipseq.tsvIf the data set contains technical replicates, eachtechnical replicateis a sample.
Otherwise, eachbiological replicateconstitutes a sample.Column names of this file must correspond to those specified in theconfig yamlfile.The sampletable must contain thesample_colcolumn.The sampletable must contain theR1column from thefile_colssection of the config.yaml.
Ifis_paired_endisTrue, then the sampletable must also contain theR2column.All paths specified must beabsolute paths.OutputThe tool outputs the following:File linksmd5 hashessample sectionpaired-end sectionFile linksThe tool creates symlinks to raw or processed files specified in the sampletable. These
are named using the format:<sample_col>_<file_colskey>.<extension>. So, for example, ifsamplenameiswt_1extension is.fastq.gzfile_colskey isR1Then the symlinked output file iswt_1_R1.fastq.gz.Note:The ‘extension’ is calculated from the raw file name as the string following the first period (.)
in the basename of the file.For example, for a file namedseq_core_237_R1.fastq.gz, the extension will be.fastq.gz.However, if the file is namedseq_core_237.R1.fastq.gzthe extension will be.R1.fastq.gz.In addition, the tool also outputs the following files:md5 hashesFor each file specified in the sampletable, md5 hashes are calculated using themd5sumutility and output to a TSV with file names
in the first column and md5 hashes in the second column. Here (templates/chipseq/md5hash.tsv)
is themd5hash.tsvcreated for the example ChIP-Seq data set mentioned above.Note:The md5 hashes of the example data are identical as it was run on empty test data. In practice,
the hashes for each file will be unique.sample sectionThis is a TSV where each row contains all files corresponding to a particular sample
including metadata columns, technical replicates and processed files if any. This can be used
to populate theSample sectionin the GEO submission template. Here (templates/chipseq/sample_section.tsv)
is thesample_section.tsvcreated for the same example ChIP-Seq data set.paired-end sectionThis is only output for PE data and lists Read 1 and Read 2 fastq files for each sample
in two columns. This can be used to populate the finalPaired-endsection in the
GEO submission template. For example, for an example RNA-Seq data set, this
is the output tsv:templates/rnaseq/paired_end.tsvExample outputHere we give examples of output produced by the tool using the example files included in thetemplates/directory.RNA-Seq dataConfig file:templates/config-rnaseq.yamlSampletable:templates/sampletable-rnaseq.tsvOutput directory (say,geo-project) will look like this:geo-project\
  ├─ wt-1-1_R1.fastq.gz -> /data/rnaseq/raw/wt-1-1_R1.fastq.gz
  ├─ wt-1-1_R2.fastq.gz -> /data/rnaseq/raw/wt-1-1_R2.fastq.gz
  ├─ wt-2-1_R1.fastq.gz -> /data/rnaseq/raw/wt-2-1_R1.fastq.gz
  ├─ wt-2-1_R2.fastq.gz -> /data/rnaseq/raw/wt-2-1_R2.fastq.gz
  ├─ wt-2-2_R1.fastq.gz -> /data/rnaseq/raw/wt-2-2_R1.fastq.gz
  ├─ wt-2-2_R2.fastq.gz -> /data/rnaseq/raw/wt-2-2_R2.fastq.gz
  ├─ md5hash.tsv
  ├─ sample_section.tsv
  └─ paired_end.tsvOutput files:templates/rnaseq/md5hash.tsvtemplates/rnaseq/sample_section.tsvChIP-Seq dataConfig file:templates/config-chipseq.yamlSampletable:templates/sampletable-chipseq.tsvOutput directory:geo-project\
  ├─ wt-1-1_R1.fastq.gz -> /data/chipseq/raw/wt-1-1_R1.fastq.gz
  ├─ wt-1-2_R1.fastq.gz -> /data/chipseq/raw/wt-1-2_R1.fastq.gz
  ├─ wt-2-1_R1.fastq.gz -> /data/chipseq/raw/wt-2-1_R1.fastq.gz
  ├─ wt-2-2_R1.fastq.gz -> /data/chipseq/raw/wt-2-2_R1.fastq.gz
  ├─ wt-1_peaks.bed -> /data/chipseq/wt-1/peaks.bed
  ├─ wt-2_peaks.bed -> /data/chipseq/wt-2/peaks.bed
  ├─ wt-1.bigwig -> /data/chipseq/wt-1/wt-1.bigwig
  ├─ wt-2.bigwig -> /data/chipseq/wt-2/wt-2.bigwig
  ├─ md5hash.tsv
  └─ sample_section.tsvOutput files:templates/chipseq/md5hash.tsvtemplates/chipseq/sample_section.tsvSingle-cell dataConfig file:templates/config-sc.yamlSampletable:templates/sampletable-sc.tsvOutput directory:geo-project\
  ├─ wt-1.bam -> /data/pi/project/data/wt-1.bam
  ├─ wt-2.bam -> /data/pi/project/data/wt-2.bam
  ├─ wt-1_features.tsv.gz -> /data/pi/project/data/wt-1_features.tsv.gz
  ├─ wt-1_barcodes.tsv.gz -> /data/pi/project/data/wt-1_barcodes.tsv.gz
  ├─ wt-1_matrix.mtx.gz -> /data/pi/project/data/wt-1_matrix.mtx.gz
  ├─ wt-2_features.tsv.gz -> /data/pi/project/data/wt-2_features.tsv.gz
  ├─ wt-2_barcodes.tsv.gz -> /data/pi/project/data/wt-2_barcodes.tsv.gz
  ├─ wt-2_matrix.mtx.gz -> /data/pi/project/data/wt-2_matrix.mtx.gz
  ├─ md5hash.tsv
  └─ sample_section.tsvOutput files:templates/single-cell/md5hash.tsvtemplates/single-cell/sample_section.tsv