TITLE: Titan-gc SARS-CoV-2 Strain Characterization Workflow for Bioconda
AUTHORS: Michelle  Su, Robert  A Petit III, Technical  Outreach and Assistance for States Team

[DESCRIPTION]
This protocol covers the process of using Titan on the command-line, which was developed by Robert Petit at Wyoming Public Heath Laboratories. Some laboratories have the computing infrastructure and capacity to conduct local bioinformatic analyses and wish to use the Titan workflow, but not on the Terra platform. Local implementation of Titan-gc eliminates bottlenecks such as data upload to the cloud as well as offers users the ability to incorporate the process into scripts if they desire. Provided steps include how to install dependencies via Conda, create input files for the workflow, and run the pipeline directly or using a Nextflow wrapper. Basic troubleshooting tips are also discussed.

For technical assistance, please contact: TOAST@cdc.gov

[STEPS]
SECTION: Set up conda environment
1. The Titan workflow and its dependencies can be installed using the Conda package manager and the Bioconda channel for bioinformatics software. To install Conda, follow the instructions in Step 1.1. To update an existing Conda installation, go to Step 1.2. To add the Bioconda channel, go to Step 1.3.

A Bioconda recipe for titan-gc is available at: https://bioconda.github.io/recipes/titan-gc/README.html


With Conda running and an activated Bioconda channel on your machine, create a new Conda environment:
 

This creates a new environment named titan-cli in which to install the titan-gc software. This Conda environment can be activated and deactivated as needed. When active, the environment name appears in parentheses to the left of the prompt. When deactivated, the normal prompt returns. For example: 
 

With the titan-cli environment activated, install and update the titan-gc software:


SECTION: Set up conda environment
1.1. Miniconda is a lightweight implementation that includes Conda and Python.
Detailed installation instructions can be found at: https://conda.io/projects/conda/en/latest/user-guide/install/index.html. 
A Conda tutorial can be found at: https://jvhagey.github.io/Tutorials/mydoc_Installation.html.

 

 

 

SECTION: Set up conda environment
1.2. Conda itself can be updated with command:
 

SECTION: Set up conda environment
1.3. Add the Bioconda channel, and required dependencies in the specified order, to an existing Conda installation with commands:
 

SECTION: Create the input file
2. Analyzing sequences with the titan-gc workflow requires an input file defining:

Same name
Workflow
Sequencing read file locations
Primer bed file location

ARTIC primer bed files are available at: https://github.com/artic-network/primer-schemes/tree/master/nCoV-2019 

The input file is created using the included titan-gc-prepare.py script in the Conda environment:
 
  
The script takes three required positional inputs (in this specific order): fastq path, workflow, and primer bed file. The script infers sample name from the sequencing read fastq filenames.

FASTQ_PATH: relative or absolute path to directory containing fastq files
Optional parameters to modify how the script looks for and processes fastq files:
        -f, --fastq_ext: .fastq.gz file extension is removed to infer sample name
        --fastq_separator: underscore is used to infer where the sample name is. i.e. _1 or _2 is removed 
        --fastq_pattern: .fastq.gz read files are looked for. If you want to change it to look for fastqs, you would put '*.fastq'
      (do not forget the single quotes or the wildcard will expand and substitute the files names)
        --pe[1/2]_pattern: specifies how to differentiate between left and right reads. Accepts R1/2, r1/2, 1/2, A/B, a/b 
        -r, --recursive: look not only in current folder for reads files but all folders in this folder

WORKFLOW: What workflow to run
Options: clearlabs, illumina_pe, illumina_se, ont

PRIMER: relative or absolute paths accepted for location of primer bed file

Optional parameters:
        --pangolin_docker: specify docker tag for version of pangolin you want to run; not compatible with --tsv
        --params: relative or absolute paths accepted for location of params.json; not compatible with --tsv
        example params.json: default params
        --tsv: this specifies the output file format. Default is the JSON file format. JSON is used to run the titan-gc directly, and TSV is used for Nextflow
 


Note: samples prepared together will be run with the same workflow. If you need to run different workflows, prepare separate input files for each.

Example command: 
 

Example JSON input file creation:

Note that the "." (period) in the first line of the example below indicates that the current directory should be used as FASTQ_PATH. 
 

Example TSV input file/FOFN (File of file names) creation:
 



SECTION: Running the titan-gc pipeline
3. The titan-gc pipeline can be run two different ways:

Directly (Step 3.1)
With Nextflow (Step 3.2)

Differences between the two implementations are primarily related to call-caching (how samples are submitted to the pipeline), job submission, and STDOUT.

Call-caching: The Nextflow wrapper offers the benefits of call-caching, essentially allowing failed runs to be resumed without rerunning finished analyses. This saves time and compute resources. Running the pipeline directly (Step 3.1) uses cromwell run, and call-caching is only available from Cromwell's server mode, which will retry failed processes 10 times before the workflow is aborted.

Job submission: The Nextflow wrapper individually submits each sample to the Titan workflow. If one sample fails, downstream samples are not affected. Alternatively, when running the pipeline directly (Step 3.1), failed samples can cause the entire workflow to abort before remaining subsequently queued samples can be run to completion. 

STDOUT: The Nextflow wrapper outputs little information, but provide a summary message to report the number of samples finished before merging results. Running the pipeline directly can output all of Cromwell's very extensive run logs to STDOUT.
 

SECTION: Running the titan-gc pipeline
3.1. Running pipeline directly:
 
 
The command takes two required inputs: input JSON file and output directory
    -i: relative or absolute path to input JSON file
    -o: relative or absolute path to output directory

Optional parameters: 
    --profile: what container software to use. 
        options: docker(default), singularity. 
    --options: provide JSON file to modify cromwell run attributes
        More info about config options can be found at https://cromwell.readthedocs.io/en/stable/Configuring/
    --quiet: Silence output from running the pipeline

Example command:
 
The running pipeline should produce extensive text output of Cromwall's run information.

If a sample fails, because Cromwell provides no call-caching, there are no options except to rerun the sample. If the input.json file included multiple samples some may have successfully completed before the workflow aborted and a new input.json should be prepared for rerunning only the failed samples.

SECTION: Running the titan-gc pipeline
3.2. Acquire the Nextflow wrapper here: https://gitlab.com/steamboat/titan-nf/-/tree/main.

DISCLAIMER: The titan-nf Nextflow wrapper is not officially supported by the Titan development team and is a separate activity supported by Robert Petit in Wyoming. This workflow is under active development and frequent updates are expected.

First, pull the Nextflow wrapper, which is not included in the Conda installation. Nexflow itself should already be installed by default. This step only needs to be completed once or whenever the Nextflow wrapper is updated. The workflow will be stored at "$HOME/.nextflow/assets/steamboat/titan-nf". 

Note: execute while your Conda environment is active unless your system has Nextflow.
 

Running the Nextflow wrapper:
 
 
The command takes one required input: input TSV file
    --samples: relative or absolute paths taken for input TSV file (created using --tsv option of titan-gc-prepare.py)

Optional parameters:
    --titan_opts: relative or absolute paths accepted for location of JSON parameter file
        example params.json: default params
    --pangolin_docker: specify docker tag for version of pangolin you want to run
    --profile: what container software to use.
        options: docker, singularity(default).
    --outdir: directory in which output folder is created. Default is the current working directory.
    --run_name: name of output folder. Default is titan-gc.

Example command:
 

Example of running nextflow wrapper:
 
 
If a sample fails, use the -resume option to retry the sample beyond the initial number of Nextflow attempts (default: 5 times). Some failures are not reproducible, and simply rerunning may solve the issue. Completed samples will not be rerun, thanks to call-caching, so the same input file can be reused without excluding the completed sampled.

Read more about Nextflow -resume at: https://www.nextflow.io/blog/2019/demystifying-nextflow-resume.html
 
Example of -resume option in Nextflow:
 


SECTION: Output
4. The output directory will include a structure similar to that shown below. Running the titan-gc pipeline using the Nextflow wrapper produces an additional "nf-info" folder. Read more about the Nextflow outputs at: https://www.nextflow.io/docs/latest/tracing.html 
 

SECTION: Troubleshooting
5. There are two places to investigate when troubleshooting:

Cromwell STDOUT: the STDOUT messages can indicate which step failed and where in the working directory to look
Cromwell working directory: this holds the STDOUT and STDERR log files for each task run

Cromwell STDOUT:
Running pipeline directly: cromwell-stdout.txt will be in your specified output directory
Running Nextflow wrapper: cromwell-stdout.txt will be in the titan-gc folder in the unique hash identifier folders in the work folder (which is by default in the directory you ran the pipeline from), e.g.:
 

Cromwell work directory:
Running pipeline directly: There will be a cromwell-executions folder (which is by default in the directory you ran the pipeline from) which contains a titan-gc folder which contains unique hash identifier folders, e.g.:
 
Running nextflow wrapper: In the nextflow work folder under the unique hash identifier, there will be a cromwell-executions folder which contains a titan-gc folder which contains unique hash identifier folders, e.g.: