Package: ncbi-submit
ncbi-submitSubmitting data to public databases is super important for publically funded laboratories, but it is not always a quick or intuitive process.ncbi-submitprovides a simple and repeatable way to upload programmatic submissions to NCBI's SRA and GenBank with shared or unique BioProjects and BioSamples. Data can be uploaded as XML or zip files to either the Test or Production environments, and once there, the reports produced by NCBI can be analyzed to check on submission status and get BioSample accessions.Installation:To install from PyPI in a virtual environment.venv:python3-mvenv.venv
..venv/bin/activate
pipinstallncbi-submitTo install from conda (not yet set up) in a new environmentncbi:condacreate-nncbincbi-submitTestingAdd NCBI credentials to file./.login_credentialsor edit them in either:./example/test.shor./config/config.pyTo test creating all example files, run:./example/test.shThis script ^^^ could also be a good starting point for your own NCBI submission pipelines. Note: There are several blocks of code in there can be commented in/out, as needed.Usagencbi_submitis intended for use on the command line, but the classncbi.NCBIcan be imported and used within custom python scripts.There are three main actions the script can do:file_prep:Prepares .tsv & .xml files for SRA, BioSample, & BioProject submissionsUsed to prepare all files for initial submission to NCBITo add in biosample accessions and prepare for GenBank submission, include the flagprep_genbank:Prepares .zip, .sbt, & .tsv files for GenBank SubmissionUsed to add BioSample accessions from a BioSample submission for a GenBank submissionftpsubmission or checkup:Interacts with NCBI's ftp host to do either of the following:submitdata to NCBI databasescheckon previous ftp submissionsget-accessionsfrom all previous ftp submissionsexample:Writes out example files for one or both of:config.py file (tellsncbi_submitlots of important info)template.sbt (used for genbank submission)SetupThe required parameters vary by which of the above actions you're attempting but at minimum require aplateandoutdir. To limit the number of parameters required via command line, aconfigfile must be used. When running from the command line, one of the three actions (file_preporftp) must be specified. With python, these are associated methods you may use on a single NCBI object.Get exampleconfig.pyfile:ncbi_submitexample--config--outdir"nbci"Python instantiation (not needed on command line):Note: This is the minimum required info for preparing data. Other parameters may be necessary for more functionality or other tasks.fromncbi_submitimportncbi_submitncbi=ncbi_submit.NCBI(fastq_dir=myFastqDir,seq_report=mySeqReport,plate=myPlate,outdir=myOutdir,config_file=myConfig,)ncbi.write_presubmission_metadata()File PreperationShell:ncbi_submitfile_prep\--test_mode--test_dir\--config"${NCBI_CONFIG}"\--seq_report"${SEQ_REPORT}"\--primer_map"${PRIMER_MAP}"\--primer_scheme"${SCHEME_VERSION}"\--outdir"${NCBI_DIR}"\--gisaid_log"${GENERIC_GISAID_LOG//PLATE/$PLATE}"\--fastq_dir${FASTQS}\--plate"${PLATE}"Python:ncbi.write_presubmission_metadata()File SubmissionShell:ncbi_submitftpsubmit\--dbbs_sra\--test_mode--test_dir\--config"${NCBI_CONFIG}"\--outdir"${NCBI_DIR}"\--fastq_dir"${FASTQS}"# wait a while and try this to download reports and view submission statusncbi_submitftp\--check--dbbs_sra\--test_mode--test_dir\--config"${NCBI_CONFIG}"\--outdir"${NCBI_DIR}"Python:ncbi.submit(db="bs_sra")# wait awhile and try this to download reports and view submission statusncbi.check(db="bs_sra")GenBank submission(NOTE: not fully tested)
To link your fasta in GenBank to the associated reads, you'll want to add in the BioSample accessions before submitting.Acquire BioSample accessions via one of these methods:download accessions.tsv file from NCBI and then usencbi_submit(Do this if you submitted to BioSample via NCBI's Submission Portal)usencbi_submitfor everything(Do this to avoid manual uploads via NCBI's Submission Portal)Shell:# dowload report.xml files to get accesssions fromncbi_submitftpcheck\--db${DB}\--outdir"${NCBI_DIR}"\--config"${NCBI_CONFIG}"\-u"${ncbi_username}"\-p"${ncbi_password}"\--plate"${PLATE}"\--fastq_dir"${FASTQS}"# add accessions to genbank.tsvncbi_submit--prep_genbank\--outdir"${NCBI_DIR}"\--config${NCBI_CONFIG}\--fasta"${GENERIC_CONSENSUS//PLATE/$PLATE}"\--plate"${PLATE}"# submit to GenBank (NOTE: db='gb')ncbi_submitftpsubmit\--dbgb\--test_mode--test_dir\--config"${NCBI_CONFIG}"\--outdir"${NCBI_DIR}"\--fastq_dir"${FASTQS}"Python:# dowload report.xml files to get accesssionsncbi.check(db="bs_sra")# prepare genbank submission files and submitncbi.submit(db="gb")## or# files can also be prepared without submitting via:ncbi.write_genbank_submission_zip()Check Submission StatusWait awhile (10+ minutes) for NCBI to start processing the submission. Then run this to download reports and view submission status.
This works for whichever db you want to check on. If not specified, you'll get results on all submitted dbs.Shell:# check GenBank submission status (NOTE: db='gb')ncbi_submitftpcheck\--dbgb\--test_mode--test_dir\--config"${NCBI_CONFIG}"\--outdir"${NCBI_DIR}"Python:# check GenBank submission status (NOTE: db='gb')ncbi.check(db='gb')Download all reports and get accessionsTo acquire the accessions for all samples submitted via ftp under your group's account,ncbi_submitcan download all xml report files and parse out the accession details. A directory will be created inoutdircontaining all submission-specific directories, each containing its report files. The-for--filesflag allows the use of a list of report files to parse. If provided, those files will be parsed for accession details rather than downloading the latest report files. The database can be specifed to indicate which accessions are desired and yield csvs (for the BioProject associated with your currentconfigfile) at<outdir>/accessions_<bioproject>.csvwith the following fields:databasefields'bs_sra'sample_name, BioSample, SRA'bs'sample_name, BioSample'sra'sample_name, SRAGet accessions by downloading report.xml filesShell:ncbi_submitftpget-accessions\--db"bs_sra"\--config"${NCBI_CONFIG}"\--outdir"${REPORT_DIR}"\-u"${ncbi_username}"\-p"${ncbi_password}"\Python:ncbi.get_all_accessions(db="bs_sra)Get accessions from list of report.xml filesShell:ncbi_submitftpget-accessions\--db"bs_sra"\--config"${NCBI_CONFIG}"\--outdir"${REPORT_DIR}"\-u"${ncbi_username}"\-p"${ncbi_password}"\-fs1/report.xmls2/report.xmlPython:ncbi.get_all_accessions(db="bs_sra",report_files=["file1","file2"])Updating samples that have already been submittedFastq read updatesIf you want to update the reads for a sample you've already submitted, you must do the followind:Emailnlm-support@nlm.nih.govand supply them with a list of SRA runs to suppress.Once suppressed, you can upload a new version of the sample where thesubmission.xmlreferences the BioSample (rather than submitting a new BioSample block) andhas a new, unique SPUID for the SRA action block.Thesubmission.xmlcan be prepared as shown below and then submitted as discussed previously inFile Submission. Whereas normally an error would occur if a previously-submitted sample appears in theseq_reportfile, the flag--update_readstellsncbi_submitto search for BioSasmple accessions of and include previously-submitted samples in thesubmission.xml. In most cases, if you are updating reads for a sample, a new SRA spuid is required. The--spuid_endingsflag takes a parameter mapping samples that are being updated to a suffix. For any explicitely names samples, the suffix(es) will be added at the end of the automatically-generated SPUID. Usually '2' is a good suffix choice (unless another update has already been made using that same suffix for the sample of interest).Other metadata updatesThese are not currently supported but could be added in the future if they seem important/useful.Shell:ncbi_submitfile_prep\--config"${NCBI_CONFIG}"\--seq_report"${SEQ_REPORT}"\--outdir"${NCBI_DIR}"\--fastq_dir${FASTQS}\--plate"${PLATE}"\--update_reads\--spuid_endings'suffix1:samp1,samp2;suffix2:samp3'Python:ncbi.write_presubmission_metadata(update_reads=True,spuid_endings={"sample1":"suffix1","sample2":"suffix1","sample3":"suffix2"})Input PathsRequired Files:config: Contains preset values and details about your lab, team, and submission plans that are necessary for submission.seq_report: Main metadata file with sample details - can be equivalent to NCBI's BioSample TSV for use with the Submission Portal.Optional Filesexclude_file: Contains a list of "sample_name"s to exclude from NCBI submission (each one on a new line).barcode_map: Used as a cross-reference. If all samples frombarcode_mapappear inseq_report, that's great. Otherwise, you'll get a warning with directions for adding samples to theexclude_fileif they shouldn't be submitted. File should have no headers. Lines must be: "{barcode}\t{sample_name}".Sometimes Required Pathsfastq_dir: Required forfile_prepandftpif submitting reads to SRA. Indicates where the fastqs should be gathered from. Any fastqs with "sample_name" values that aren't supposed to be submitted will be ignored.outdir: Highly recommended but will defualt to "./ncbi" or "./ncbi_test". A directory to house output (submission reports,exclude_file, output fromfile_prep). Will be created, if needed.subdir: Only used forftptasks. A prefix to use for submissions for the given dataset. Defaults toplate, if plate is provided.Links to xml template examples/schema:File typeBioProjectBioSampleSRAGenBankDescription/LinkWebpage✓✓✓✓Protocols & TSVs for use at Submission PortalXMLcreatecreatecreateSRA submission w/ new BioSample & BioProjectXMLlinkcreatecreateSRA submission w/ new BioSample & existing BioProjectXMLlinklinkcreateSRA submission w/ existing BioSample & BioProjectXMLcreateGenBank XMLdocexampleExample GenBank submission zipXSDschemaBioSample XML SchemaXSDschemaBioProject XML SchemaerrvalidateSubmission Error Explanations