TITLE: Titan Illumina PE SARS-CoV-2 Strain Characterization Workflow for the Terra Platform
AUTHORS: Jill V Hagey, Kevin  Libuit, Frank J Ambrosio, Technical  Outreach and Assistance for States Team

[DESCRIPTION]
The Titan_Illumina_PE workflow is a part of the Public Health Viral Genomics Titan series for SARS-CoV-2 genomic characterization. Titan_Illumina_PE  was written specifically to process Illumina paired-end (PE) read data. Input reads are assumed to be the product of sequencing tiled PCR-amplicons designed for the SARS-CoV-2 genome. The most common read data analyzed by the Titan_Illumina_PE workflow are generated with the ARTIC V3 protocol. However, alternative primer schemes such as the QIAseq Primer Panel are also suitable for this workflow. The primer sequence coordinates of the PCR scheme utilized must be provided in BED format along with the raw Illumina read data. Upon initiating a Titan_Illumina_PE job, the input primer scheme coordinates and raw paired-end Illumina read data provided for each sample will be processed to perform consensus genome assembly, infer the quality of both raw read data and the generated consensus genome, and assign lineage or clade designations as outlined in the Titan_Illumina_PE data workflow diagram below.

Additional technical documentation for the Titan_Illumina_PE workflow is available at:
https://public-health-viral-genomics-theiagen.readthedocs.io/en/latest/titan_workflows.html#titan-workflows-for-genomic-characterization


   
Required input data for Titan Illumina PE:
Illumina paired-end read data (forward and reverse FASTQ files per sample)
Primer sequence coordinates of the PCR scheme utilized in BED file format

Video Instruction:
Theiagen Genomics: Titan Genomic Characterization
https://www.youtube.com/watch?v=zP9I1r6TNrw

Theiagen Genomics: Titan Outputs QC
https://www.youtube.com/watch?v=Amb-8M71umw

For technical assistance please contact us at: TOAST@cdc.gov


[STEPS]
SECTION: Setup Terra and Google Cloud Accounts
1. 
 

 

 


SECTION: Import Titan Illumina PE workflow from Dockstore
2. Importing the Titan Workflow from Dockstore to the User Workspace

We will first walk through step by step how to import the Titan workflow and at the end there is a video showing the full process.

First create a new workspace by clicking on the three parallel lines in the top left-hand corner, followed by clicking the 'Workspaces' tab then click the "+" sign on the top right of the newly loaded page. Follow the directions to link the workspace to your billing project. The video at the end shows all steps involved. DO NOT INCLUDE SPACES IN THE WORKSPACE NAME, use underscores instead. Spaces will cause an error later in the pipeline.

  
The script for the Titan workflow is located in the Dockstore repository and has to be imported into the user Terra Workspace. There are two ways to get to this page and the video shows both. The first method starts by clicking on the three parallel lines in the top left-hand corner, followed by clicking the 'Library' tab and finally click the 'Workflows' button.

  
In the 'Workflows' panel, under 'Find Additional Workflows' click on the 'Dockstore' link in the grey box on the right side of the page.

  
On the left side of the Dockstore page, search for 'theiagen/public_health_viral_genomics/Titan_Illumina_PE' in the search bar.

  
Click the 'theiagen/public_health_viral_genomics/Titan_Illumina_PE' link, which will take you to a new page. 

  
On the right-hand side of the page under the 'Launch with' window, click the 'Terra' button. It should bring you back to the Terra platform within the 'Import Workflow' page. If you already have a workspace created you can simply import it into that workspace. If not, see step 2.1 to create a new workspace.

    
After clicking the 'import' button, you should be automatically directed to the Titan workflow panel in the new workspace page that was just created. 

Make sure you pick the latest version (or an older stable version you want to use) from the drop down version menu before continuing. The "main" and "dev" versions are under active development and aren't stable!

  
 Video of the entire process:
 

SECTION: Upload Fastq Sequence Files
3. There are three options for uploading your files: 
    1. Upload on Terra (can only do single files).
    2. Upload to Google bucket and link to Terra (can do single files or bulk files/folders).
    3. Upload via 'https://app.terra.bio/#upload' -- This is the easiest option
SECTION: Running the Titan Illumina PE Workflow
5. To run the Titan workflow, click on the 'Workflows' panel in the newly created workspace. It should bring you to the following page:

  
Click on the 'Titan_Illumina_PE' tile and it will take you to a new page. Double check that you are using the latest version of the workflow. Alternately, you may specify another version, but should only pick a stable version (with numbers) NOT a 'main' or 'dev' version. 

   
Click the 'Run workflow(s) with inputs defined by data table' option button and then in the 'Select root entity type' pull-down menu, select 'sample' or whichever entity type you specified in the sample table. The first four rows in the 'Inputs' tab should each have a yellow caution icon. These four rows represent variables that have to be provided by the user. This is the information that we populated the sample data table with in the previous steps. 

Make sure "Use call caching" is checked and then select the root entity type for the data you wish to analyze (here it's sample).

NOTE: Call caching allows Terra to identify and skip jobs that have been run previously; this option is by default enabled to  avoid unnecessary compute costs. More information on Terra call caching, including examples of when you may want to disable this feature, is available through the Terra Support Documentation. 

 
For the first row, click on the 'Attribute' text box and then click 'workspace.Artic_V3_nCov2019_primers' (or whatever you named the primer bed file). For the next row, the 'read1_raw' variable, click on the 'Attribute' text box and then click 'this.Forward_Read'. In the third row, the 'read2_raw' variable, click on the 'Attribute' text box and then click 'this.Reverse_Read'. Finally, in the third row, the 'samplename' variable, click on the 'Attribute' text box and then click 'this.sample_id'. Click the 'Save' button on the top right-hand side of the page. The yellow caution icons should disappear.

    
 


Next, click the "output" tab and select "use defaults" to autofill in the names of the output files according to the sample names. If you forget to select this option, you will have trouble accessing the results! Save changes by clicking the 'Save' button.

  
You are now ready to run the Titan_Illumina_PE workflow! Click the blue 'select data' button and select the samples you want to run or click the check box at the very top left to select all of them to be run. 

   
Click on the 'Run Analysis' button to the right of the 'Outputs' tab. A popup window should appear titled 'Confirm launch'. If the 'Run Analysis' button is greyed out, you need to save your recent changes by clicking the 'Save' button.

     
Click the 'Launch' button to access the 'Job History' panel where each sample will be queued for the Titan_Illumina_PE analysis.

  
The status will change from 'Queued' to 'Submitted' to 'Running.' After the workflow finishes, the status column will indicate either 'successful' or 'failed.' 

Video of entire process:

This starts by loading the primer bed file, which we did in the last step, but here we show an alternative method to upload it to your workspace. After uploading, you will likely have to reload the page before it shows up as an option on the workflows page (as shown in the video). 
 


SECTION: View and Download the Titan Output Report
6. 
First, verify that all of the samples have completed the analysis run by looking at the 'Workflow Statuses' section in the top left of the 'Job History' panel. The job has completed when all the samples have a status of 'Succeeded' with a green checkmark. 

   
Then go to the 'Data' panel and under the 'Tables' tab click on the sample table that you created and uploaded in step 4. It will be named 'sample (#)' where # is the number of samples in your file. 

     
The Terra sample table will now include additional attributes added by the workflow, according to your specified output names (set to default in this example). You can reduce the number of visible fields by clicking the gear icon at top right. Select only the fields you want to see then click "Done".


  
To download the consensus sequence for a particular sample, click on the link in the 'assembly_fasta' attribute column and a popup window titled 'File Details' should appear. Click the 'Download For < $0.01' button, which will download the consensus sequence to your local directory. This can be uploaded to NextClade to visualize if desired (see step 10).


 
Finally, the NextClade and PANGO lineage assignments can be viewed in the Terra sample report by scrolling to the right. You can download or copy this report by using either the 'Download All Rows' or 'Copy Page To Clipboard' buttons at the top of the table. 

 
An explanation of each output column is available here:
https://public-health-viral-genomics-theiagen.readthedocs.io/en/latest/titan_workflows.html#outputs

And also described in the video here:
https://www.youtube.com/watch?v=Amb-8M71umw&list=PLU47xRg_MKJrtyoFwqGiywl7lQj6vq8Uz&index=4 

 

SECTION: Metrics for Troubleshooting
8. FastQC Raw Output
These data are the number of raw reads in an input file and indicates how much sequencing occurred.

 
We can see that for the second sample a large number of reads were lost during the QC process, as we started with ~58k reads and only ~3k passed QC filtering parameters. We lost roughly half of the reads in the other two samples, but this might be fine if other parameters indicate that a quality consensus sequence was still recovered. 

Percent of Reads Identified to be Human and SARS-CoV-2
These data indicate how much human read contamination is in the sequencing reads. Ideally, the number of reads containing human data should be as low as possible and the percent of reads assigned to SARS-CoV-2 should be high. If the sequencing run contains greater than ~15% human reads, the concentration of SARS-CoV-2 in your original sample may be too low to generate sufficient coverage depth for a good quality assembly, or there is an issue in the wet lab protocol that is leading to the low generation of SARS-CoV-2 reads. 

 
The first and second sample (rows 1 and 2) are fairly free of contamination with 97.97% and 84.67% of the reads sequenced belonging to SARS-CoV-2, respectively. However, in the third sample there is a higher amount of contamination with 2.81% of reads belonging to human and only 52.73% of reads determined to be of SARS-CoV-2 origin. Thus, there is roughly ~44% of reads that are some other source of contamination that is not human. If a lab is having consistent problem with contamination that is not identified to be of human origin with this workflow, a larger Kraken database can be used to determine the offending organism. Contact TOAST@cdc.gov if you would like assistance with this or read Kraken2's documentation for how to do this on your own. Users should refer to their lab's cut-offs thresholds for quality metrics to determine if a sample requires resequencing.  

Bam File
This file is generated during alignment and contains all information regarding the alignment to the WuHan-1 reference. This file is found in the 'aligned_bam' column and we can download this file just like we did with the consensus sequence in step 6. These files can then be visualize this with Geneious, CLC workbench or IGV. 

Variants File - This is a text file that contains information regarding variants that were identified in the consensus assembly relative to the Wuhan-1 reference. This file is found in the 'ivar_tsv' column and can download this file just like we did with the consensus sequence in step 6. 

  
If you look at the second row, this tells us that at position 1059 in the reference genome there was a "C" in the sequence that was run there was a "T" instead. You can also download and check the file in the column 'nextclade_tsv' that has specific information about exact mutations in this sample. 

For Pro Users:

If for some reason you want to dig into intermediate files, you can go to the "JOB HISTORY" tab and click on the "execution directory" icon in the "Links" column. This will take us to a directory of folders that contain the output for each task.

  
As an example we will look into into the Trimmomatic output from the sample that lost a lot of reads during the filtering process. 
 

SECTION: Reviewing Quality Metrics
7. At the end of this process, we should ideally have a complete consensus assembly sequence, which represents the entirety of the RNA sequence present in the virion in the sample. However, a full-length sequence is often not generated at the end of the protocol, but rather fragments of the genome. We can look at metrics to determine how fragmented our genome is relative to the reference sequence. There are 3 metrics in our report to help us assess this. 

Assembly length unambiguous - the final count of ATCGs (i.e. unambiguous bases) within the consensus assembly. 
A higher number here will indicate a more complete consensus sequence. 

  
Number of Ns - the converse of assembly length unambiguous - the final count of the number of Ns (i.e. ambiguous) bases. These are completely ambiguous bases due to either too little signal or too much noise for the basecaller to confidently determine what base was at this position. 
If there was an "S" (basecaller couldn't tell if it was an C or G) or "W" (basecaller couldn't tell if it was an A or T) these counts are considered "semi-ambiguous" and are not included in either the number of N or assembly length unambiguous metrics. 
A higher number of Ns will mean you have a more fragmented assembly while fewer Ns indicates a strong consensus assembly

Percent reference coverage - portion of the genome covered by consensus assembly. This uses the assembly length unambiguous metric to gauge how much of the reference sequence (Wu Han-1) is covered by the consensus assembly. 
Takes the #ATCG/length of SARS-CoV-2 reference genome (29903bp)*100
Ideally this should to be 100%, which would mean the assembly covered 100% of the length of the reference genome unambiguously

   
Here we see that the second sample has ~11k base pairs that are completely ambiguous. This has lead to only 62.33% of the reference genome being represented by this sequencing run. This indicates that the consensus sequence generated was highly fragmented. Further, depending on where this fragmentation occurred, we might not be able to trust the NextClade and PANGO lineage assignments. In contrast, the other two samples have a low number of Ns and show >97% coverage of the reference genome, suggesting these consensus sequences are fairly complete and the lineage assignments are likely to be correct if the fragmented areas are not in key regions (i.e. the spike protein). Users should refer to their lab's cut offs for quality metrics to determine if a sample requires resequencing.  

If a lab is having consistent problem with contamination that is not identified to be of human origin with this workflow a larger Kraken database can be used to determine the offending organism. Contact TOAST@cdc.gov if you would like assistance with this or read Kraken2's documentation for how to do this on your own. 
SECTION: Upload Fastq Sequence Files
3.1. Upload fastq sequence files one at a time to be used in the analysis.

Click on the 'Data' panel in the newly created workspace and then click on the 'Files' tab

 
Under the 'Files' tab, you can either drag-and-drop your files into this space or move the mouse over the blue plus sign icon in the bottom right-hand corner and click 'upload'. Upload the sequence files you'll need for this analysis. Each fastq sequence file must be uploaded individually using this method. 

Once the files are uploaded, you will need to create a table to associate your files with their corresponding link to their Google Cloud location.
 

Finally, the completed Terra sample file will need to be uploaded to the newly created Titan workspace. On the right-hand side of the workspace 'Data' panel there is a 'Tables' tab. Click the blue plus sign icon on the right edge of the 'Tables' tab. A popup window should appear titled 'Import Table Data'. Select your completed tab-separated sample file for upload and then click the 'Upload' button.

  
If the upload is successful then the sample file should be located under the 'Tables' tab as 'sample (#)' where # is the number of samples in your file. 

   
Here is a video showing the process.
 

SECTION: Upload Fastq Sequence Files
3.3. To upload via the Terra Data Uploader.

Navigate to 'https://app.terra.bio/#upload'


Select the Terra Workspace to which you would like to upload your fastq files. This will be the same workspace created in the previous step.

Click the '+ Create a new collection' link and enter a name for your new collection of fastq files. DO NOT INCLUDE SPACES IN THE COLLECTION NAME, use underscores instead. Spaces will cause an error later in the pipeline.
 


  
Click the 'Create Collection' button and you will be routed to the data uploader page where you can drag and drop fastq files directly into your browser window to initiate upload.

Drag and drop the fastq files that you would like to upload in to the upload space.

  
Once your files have been successfully uploaded, select "NEXT>" to proceed to the metadata upload page.

To upload a metadata table and create a Terra sample table in your workspace, open excel and populate your spreadsheet with the root entity designation in cell A1 (here it is entity:sample_id), "Forward_Reads" in B1, and "Reverse_Reads" in C1 as headers; underneath these headers, proceed to populate each row. 

Important note on column names: DO NOT USE SPACES! As we did before in creating our workspace and collection names use "_" instead of spaces! The first column MUST have the name "entity:sample_id", you can call the other columns whatever you like, but we recommend you use something that denotes forward and reverse reads for clarity.

 
An easy way to create this file is to run the following in the directory where your fastq files are stored:
 
This will create the a file called Terra_Upload.txt for you. This assumes your files have the format string_1.fastq.gz and string_2.fastq.gz.

NOTE: In this example, our root entity type is "sample_id" so, in cell A1, we have written "entity:sample_id". A "root entity" is the smallest piece of data a workflow can use as input (blue square). We can check what is required from the workflow by looking at what fields are set to required (red rectangle). The root entity is always defined with "entity:" and must always end in "sample_id"

   
 


Save this sheet as a tab delimited file text file.

  
An example, metadata table is found below. 
 

Drag-and-drop your metadata table to the Terra #Upload page and select CREATE TABLE to complete the process.

  
Video of entire process:
 

SECTION: Upload Primer Bed Files
4. For the Titan_lllumina_PE workflow v1.2.0, you will need to upload a bed file for the primers you used. The reasons for this and a video example can be found on YouTube. This will need to be done once and then you can use the same file for every run that uses that primer set. Visit the ARTIC Network's GitHub repository to download a bed file for the ARTIC primers. 

  
Select the folder for the version of artic primers you are using. Then click on "nCoV-2019.primer.bed" and click the "Raw" button. 

  
  
Copy and paste everything from this new page to a new notepad document. DO NOTE USE MICROSOFT WORD! Save as a text file with the primer version number in the filename. Then you will need to go into the folder and change the extension from ".txt" to ".bed". If necessary, click "View" at the top of the folder window (red circle) and ensure the "File name extensions" option (red rectangle) is selected. 
 
 
Then right click on metadata file and select "rename" then change ".txt" to ".bed". 

  

Navigate to 'https://app.terra.bio/#upload'. Once on the upload page, click on the workspace you created before and create a new collection by clicking the blue "+" next to "select a collection" in the middle top of the page. Name the collection, here I will name it "Titan-Illumina-Files". 

 
Drag and drop the primer file you saved into this collection. The bed file will now show up as a hyperlink that actually links it from the Google bucket where it was actually uploaded to (recall Terra uses google cloud as a backend). Right click on this blue hyperlink and select "copy link address". 
 
  
Now we will need to add the primer bed file as a data workspace element. Navigate back to your workspace and under the "Data" tab select "Workspace Data" from the menu on the right hand side. Click the "Add Variable" blue "+" in the bottom right hand corner. In the key field, name the file something like "artic-v3-nCov2019-primers" In the "value" field paste the link address to the google bucket you copied earlier. Save the file by clicking the check mark on the far right. 

  
For the bed file to appear in your workspace, you must sign out and sign back in before continuing to the next step. 

Here is a video of the entire process:
 
 
SECTION: Import Titan Illumina PE workflow from Dockstore
2.1. Alternatively, if you have not already made a workspace, you will need to before importing the workflow. To do this, under 'Destination Workspace' click the 'create a new workspace' button. A pop-up window titled 'Create a New Workspace' should appear. Name your new workspace and associate it with a billing account using the 'Billing project' drop-down menu. The Terra Billing Account you created in the previous step should be available. Finally, click the 'Create Workspace' button.

  
After clicking the 'Create Workspace' button, you should be automatically directed to the Titan workflow panel in the new workspace page that was just created. 

Make sure you pick the latest version (or an older stable version you want to use) from the drop-down version menu before continuing. The "main" and "dev" versions are under active development and aren't stable!

  

SECTION: Submit Consensus Sequences to Public Repositories
9. If you plan on submitting the SARS-CoV-2 consensus sequence to either the GenBank or GISAID public repositories, please refer to the following documentation for submission criteria and minimum quality control thresholds.

GenBank Submission Criteria: About GenBank Submission (nih.gov)
GISAID Submission Criteria:   

The Mercury workflows are designed to prepare genome assemblies and sample metadata on the Terra platform for subsequent GISAID and NCBI submission. 

Mercury Overview:  https://www.youtube.com/watch?v=h8YASVckOrw

After preparing consensus sequences of SARS-CoV-2 and sample metadata for submission, use the following protocols submit them to the public repositories. Completing submissions in this order allows all the sequence information to be linked together.

1. GISAID submission protocol
2. NCBI submission to BioSample, and BioProject SRA protocol
3. Genbank submission protocol
SECTION: NextClade Visualization
10. After you downloading the consensus sequence for a particular sample, following directions in step 6. 

  
Visit to the NextClade website. Drag-and-drop the downloaded fasta file and the screen will automatically take you to the analysis page.

  
Hover over the different columns to get more information. Click the "Show Tree" button at the top right of the page to see where the sequence sits on the tree of other consensus sequences from SARS-Cov-2.

   
   
Video of the full process:
 




SECTION: Upload Fastq Sequence Files
3.2. You will likely have many samples to upload and you can do this by going directly to your Google bucket.

First, go to “DASHBOARD” tab in your workspace and click “Google Bucket” at the bottom right corner of the same page.

  
This will direct you to your “Google Cloud Platform” page for data uploading. Click “UPLOAD FILES” in the middle of this page to upload single or multiple fastq files. Or you can click “UPLOAD FOLDER” to upload a folder with multiple fastq files stored inside. This will cost some google cloud credits (e.g.4.4 Gb for $0.75).

  
Go back to your Terra account and click “DATA” tab. The successfully uploaded fastq files will show up. If the upload is successful then the sample file should be located under the 'Tables' tab as 'sample (#)' where # is the number of samples in your file. 

    
Here is a video of the process:
 


SECTION: How to debug a failed run
11. In this example of a failed run, we will look at an error caused by adding spaces into your collection name. If there is a failure for some other reason, you can follow similar steps to understand why your job failed. 

If a run fails, you will see this indicated in the job history screen in the "status" column. 

   
To understand why it failed, click the "workflow dashboard" icon in the "links" column. This will take you to a new screen and you can click the arrows next to the "message" to see what it says. Here we see there are two errors that direct us to a log file to check. To find out more, click the "execution directory" icon under the links header. This will take you to the Google Cloud location containing the run outputs.

 
Follow the file path in the Google Cloud location to the log files that were referenced in the error messages (in red/orange text in the above photo).  

  
Click on the "Authenticated URL" link that will take you to a text file. 

 
In the text file, we can see that there was an error that cause by there being a space between "Bad" and "Sample" in our file path that was created when we made "Bad Sample" rather than "Bad_Sample" as our collection name. 


 

Video of the whole process.  
 
 
Contact TOAST@cdc.gov for assistance with error messages and debugging job failures.