Package: ena-upload-cli
ENA upload toolThis command line tool (CLI) allows easy submission of experimental data and respective metadata to the European Nucleotide Archive (ENA) using tabular files or one of the excel spreadsheets that can be found on thistemplate repo. The supported metadata that can be submitted includes study, sample, run and experiment info so you can use the tool for programmatic submission of everything ENA needs without the need of logging in to the Webin interface. This also includes client side validation using ENA checklists and releasing the ENA objects. This command line tool is also available as aGalaxy tooland can be added to you own Galaxy instance or you can make use of one of the existing Galaxy instances, likeusegalaxy.euorusegalaxy.be.OverviewThe metadata should be provided in separate tables or files carrying similar information corresponding to the following ENA objects:STUDYSAMPLEEXPERIMENTRUNYou can set the tool to perform the following actions:add: add an object to the archivemodify: modify an object in the archivecancel: cancel a private object and its dependent objectsrelease: release a private object immediately to the publicAfter a successful submission, new tsv tables will be generated with the ENA accession numbers filled in along with a submission receipt.Tool dependenciespython 3.7+ including following packages:GenshilxmlpandasrequestspyyamlopenpyxljsonschemaInstallationpip install ena-upload-cliUsageMinimal:  ena-upoad-cli --action {add,modify,cancel,release} --center CENTER_NAME  --secret SECRET

All supported arguments:

  -h, --help            show this help message and exit
  --version             show program's version number and exit
  --action {add,modify,cancel,release}
                         add: add an object to the archive
                         modify: modify an object in the archive
                         cancel: cancel a private object and its dependent objects
                         release: release a private object immediately to public
  --study STUDY         table of STUDY object
  --sample SAMPLE       table of SAMPLE object
  --experiment EXPERIMENT
                        table of EXPERIMENT object
  --run RUN             table of RUN object
  --data [FILE ...]     data for submission
  --center CENTER_NAME  specific to your Webin account
  --checklist CHECKLIST
                        specify the sample checklist with following pattern: ERC0000XX, Default: ERC000011
  --xlsx XLSX           filled in excel template with metadata
  --isa_json ISA_JSON   BETA: ISA json describing describing the ENA objects
  --isa_assay_stream ISA_ASSAY_STREAM
                        BETA: specify the assay stream(s) that holds the ENA information, this can be a list of assay streams
  --auto_action         BETA: detect automatically which action (add or modify) to apply when the action column is not given
  --tool TOOL_NAME      specify the name of the tool this submission is done with. Default: ena-upload-cli
  --tool_version TOOL_VERSION
                        specify the version of the tool this submission is done with
  --no_data_upload      indicate if no upload should be performed and you like to submit a RUN object (e.g. if uploaded was done separately).
  --draft               indicate if no submission should be performed
  --secret SECRET       .secret.yml file containing the password and Webin ID of your ENA account
  -d, --dev             flag to use the dev/sandbox endpoint of ENAMandatory arguments: --action, --center and --secret.ENA WebinA Webin can be madehereif you don't have one already. The Webin ID makes use of the full username looking like:Webin-XXXXX. VisitWebin onlineto check on your submissions ordev Webinto check on test submissions.The .secret.yml fileTo avoid exposing your credentials through the terminal history, it is recommended to make use of a.secret.ymlfile, containing your password and username keywords. An example is given in the root of this directory.ENA sample checklistsYou can specify ENA sample checklist using the--checklistparameter. By default the ENA default sample checklist is used supporting the minimum information required for the sample (ERC000011). The supported checklists are listed on ourtemplate repo.Fixed sample columnsThe command line tool will automatically fetch the correct scientific name based on the taxon ID or fetch the taxon ID based on the scientific name. Both can be given and no overwrite will be done.Mandatory:alias,title,sample_description,collection date,geographic location (country and/or sea)and eitherscientific_nameortaxon_id(preferred)Optional:common_name,sample_descriptionaliastitletaxon_idscientific_namecommon_namesample_descriptioncollection dategeographic location (country and/or sea)sample_alias_4sample_title_22697049Severe acute respiratory syndrome coronavirus 2covid-19sample_description_12020-10-11Argentinasample_alias_5sample_title_32697049Severe acute respiratory syndrome coronavirus 2covid-19sample_description_22008-01-24BelgiumViral submissionsIf you want to submit viral samples you can use theENA virus pathogenchecklist by addingERC000033to the checklist parameter. Check out ourviral example commandas demonstration. Please use theENA virus pathogenchecklist in our template repo to know what is allowed/possible in theControlled vocabularyfields.ENA study, experiment and run tablesPlease check out thetemplateof your checklist to discover which attributes are mandatory for the study, experiment and run ENA object.Dev instanceBy default the submission will be done using following url to ENA:https://www.ebi.ac.uk/ena/submit/drop-box/submit/?auth=ENAUse the--devflag if you want to do a test submission using the tool by the sandbox dev instance of ENA:https://wwwdev.ebi.ac.uk/ena/submit/drop-box/submit/?auth=ENA. A TEST submission will be discarded within 24 hours.Submitting a selection of rows to ENAThere are two ways of submitting only a selection of objects to ENA. This is handy for reoccurring submissions, especially when they belong to the same study.Manual: you can add an optionalstatuscolumn to every table/sheet that contains the action you want to apply during this submission. If you chose to add only the first 2 samples to ENA, you specify--action addas parameter in the command and you add theaddvalue to the status column of the rows you want to submit as demonstrated below. Same holds for the actionmodify,releaseandcancel.Automatic (BETA): using the--auto_actionit is possible to auto detect wether an object (using the alias) is already present on ENA and will fill in the specified action (--actionparameter) accordingly. In practice, this means that if a user chooses to add objects and we already find this object already exists using its alias, this objects will not be added. On the other hand, if the command is used to modify objects, we want to apply this solely on objects that already exist on ENA. The detection only works with ENA objects that are published and findable on the website trough the search function (both the dev and live website). If the tool does not correctly detect the presence of your ENA object, we suggest to use the more robust manual approach as described above.Example with modify as seen in theexample sample modify tablealiasstatustitletaxon_idsample_descriptionsample_alias_4modifysample_title_12697049sample_description_1sample_alias_5sample_title_22697049sample_description_2IMPORTANT: if the status column is given but not filled in, or filled in with a different action from the one in the--actionparameter, no rows will be submitted! Either leave out the column or add to every row you want to submit the correct action.Using Excel templatesWe also support the use of specific excel templates, designed for each sample checklist. Use the--xlsxcommand to add the path to an excel template file filled in from thistemplate repo.The data filesSupported dataRead dataGenome AssemblyTranscriptome AssemblyTemplate SequenceOther AnalysesMost files uploaded to the ENA FTP server need to be compressed.More information on how ENA wants to receive the files can be foundhere.Note for data upload:Uploaded files are persistently stored on the ENA server after the upload for some time.
Thus, if multiple test submission are performed, it is possible to skip the data upload with--no_data_uploadin
subsequent submissions.
This also allows uploading (large) datasets separately e.g. withaspera.
For the--no_data_uploadargument,  data file(s) still need to be provided with--dataif a RUN object is submitted without its MD5 sums in thefile_checksumcolumn.Releasing and canceling a submissionIf you want to release or cancel data, you can do so by usingcancelorreleasein the--actionparameter in the command line. Tables that have to be released or cancelled need an accession column with corresponding accession ids. This means that you first have to useaddto submit your data, and use afterwords the updated table with accession ids, if you did not yet submit your data.By default the updated tables after submission will have the actionaddedin their status column. Don't forget to change the values toreleaseorcancelif you want to use one of these actions (or delete the status column if your action applies for the whole table).NOTE: Releasing a study will make all child elements like runs and experiments public.Tool overviewinputs:metadata tables/excelsheet/isa_jsonexamples inexample_tableand on thistemplate repofor excel sheets(optional) define actions instatuscolumn e.g.add,modify,cancel,release(when not given the whole table is submitted)to perform bulk submission of all objects, thealiases idsin different ENA objects should be in the association where alias ids in experiment object link all objects togetherexperimental dataexamples inexample_dataoutputs:a receipt.xml file in the working directory with the receipt from the ENA submissionmetadata tables with updated info in the same directory of inputs:updated status:added,modified,canceled,releasedaccession idssubmission datefile checksums in runs table if not giventaxon id or scientific name in sample table if not givenTest the toolAdd metadata and sequence dataena-upload-cli --action add --center 'your_center_name' --study example_tables/ENA_template_studies.tsv --sample example_tables/ENA_template_samples.tsv --experiment example_tables/ENA_template_experiments.tsv --run example_tables/ENA_template_runs.tsv --data example_data/*gz --dev --secret .secret.ymlAdd metadata onlyena-upload-cli --action add --center 'your_center_name' --study example_tables/ENA_template_studies.tsv --sample example_tables/ENA_template_samples.tsv --experiment example_tables/ENA_template_experiments.tsv --run example_tables/ENA_template_runs_md5sums.tsv --dev --secret .secret.ymlAdd studiesena-upload-cli --action add --center 'your_center_name' --study example_tables/ENA_template_studies.tsv --dev --secret .secret.ymlModify sample metadataena-upload-cli --action modify --center 'your_center_name' --sample example_tables/ENA_template_samples_modify.tsv --dev --secret .secret.ymlViral dataena-upload-cli --action add --center 'your_center_name' --study example_tables/ENA_template_studies.tsv --sample example_tables/ENA_template_samples_vir.tsv --experiment example_tables/ENA_template_experiments.tsv --run example_tables/ENA_template_runs.tsv --data example_data/*gz --dev --checklist ERC000033 --secret .secret.ymlUsing an Excel templateena-upload-cli --action add --center 'your_center_name' --data example_data/*gz --dev --checklist ERC000033 --secret .secret.yml --xlsx example_tables/ENA_excel_example_ERC000033.xlsxUsing an ISA JSONena-upload-cli --action add --center 'your_center_name' --data example_data/*gz --dev --secret .secret.yml --isa_json tests/test_data/simple_test_case_v2.json --isa_assay_stream "Ena stream 1"Release submissionena-upload-cli --action release --center'your_center_name' --study example_tables/ENA_template_studies_release.tsv --dev --secret .secret.ymlNote for Windows users:Windows, by default, does not support wildcard expansion in command-line arguments.
Because of this the--data example_data/*gzargument should be substituted with one containing a list of the data files.
For this example, use:--data example_data/ENA_TEST1.R1.fastq.gz example_data/ENA_TEST2.R1.fastq.gz example_data/ENA_TEST2.R2.fastq.gz