TITLE: Auger/Auspice/UShER SARS-CoV-2  Cluster Detection Workflow for the Terra Platform
AUTHORS: Anusha  Ginni, Jill V Hagey, Michael  Weigand, Technical  Outreach and Assistance for States Team

[DESCRIPTION]
The Titan_Augur workflow processes concatenated assembly fasta data for SARS-CoV-2 phylogenetic analysis and cluster detection using Augur, the modular bioinformatics engine behind Nextstrain. Two workflows, Titan_Augur_Prep and Titan_Augur_Run, must be completed in series and produce the required inputs for visualization with Auspice to evaluate genetic relatedness among sequences or draw phylogenetic inference. 

Inputs for Titan_Augur_Prep include the assembled fasta, metadata (uploaded tsv), and PANGO lineage assignments for each sample. Running Titan on Terra generates the necessary assembly fasta and PANGO lineage assignments, but these data can also be uploaded if prepared using another pipeline. Titan_Augur_Prep processes these data and outputs the correctly formatted metadata table and concatenated fasta files required for subsequent Titan_Augur_Run analyses. Titan_Augur_Run then generates the inputs needed for phylogenetic inference on the Auspice or Nextstrain interactive visualization tools. 

This protocol also includes steps for visualization using Auspice as well as the UCSC UShER webportal.

For technical assistance, please contact: TOAST@cdc.gov

[STEPS]
SECTION: Setup Terra and Google Cloud Accounts
1. 
 


SECTION: Import Augur workflows from Dockstore
2. Importing the Titan_Augur Workflows from Dockstore to the Terra User Workspace

Note: The Augur workflow includes two steps: Titan_Augur_Prep and Titan_Augur_Run.

The Titan_Augur_Prep workflow prepares the metadata and assembly fasta required by the Titan_Augur_Run workflow, so both pipelines must be imported into your workspace. 

The Augur workflows are hosted on the Dockstore repository and must be imported into the user's Terra Workspace. Begin by clicking on the three parallel lines in the top left-hand corner, followed by clicking the 'Library' tab and finally click the 'Workflows' button.

  
On the right side of the page under 'Find Additional Workflows' select 'Dockstore' from the dialogue box .

   
On the left side of the Dockstore page search for 'theiagen/public_health_viral_genomics/Titan_Augur_Run' in the search bar.

  
Click the 'theiagen/public_health_viral_genomics/Titan_Augur_Run' link. This will take you to a page where you can import the workflow into your Terra workspace. 

  
On the right-hand side of the page under the 'Launch with' window click the 'Terra' button. It should bring you back to the Terra platform within the 'Import Workflow' page. If you already have a workspace select it from the dropdown menu Under 'Destination Workspace' and click 'Import'. If not you will need to create a workspace first (see note below) 
 

After clicking the 'Import' button you should be automatically directed to the Augur workflow panel in your specified workspace. 

  
Now complete the same steps to import the theiagen/public_health_viral_genomics/Titan_Augur_Prep workflow as well.

The video below demonstrates how to import both the Titan_Augur_Prep and Titan_Augur_Run workflows.
 


SECTION: Analysis Inputs/Data uploading
3. 
 

The Augur workflows require the assembled fasta sequence (consensus sequence), PANGO linage assignments, and metadata for each sample. These required inputs are listed on the workflow page: 

 
The assembly file is generated from your genomic characterization workflow. If you have run Titan workflows on Terra then you will already have generated the assembly fasta files and all you need at this point is to select the samples from your analysis sample sets within your workspace. We will do this in step 4, but will still need to upload your metadata for each sample if that is not already associated with them. 

Construct a file with the following columns that are required metadata for each sample of interest:
Collection_date	
iso_country	
iso_state	
iso_continent	
iso_county
Here is an example for TABLE file that you will upload to your workspace:  

To associate your metadata with the assembly fasta and pango lineages already in Terra, being by going to the DATA page and click the blue "+" sign next to the TABLES tab on the left of the web page. A pop-up window will appear to import a table file (see example file above) and either drag and drop the file or browse to find it. Once uploaded return to your samples and notice you will now how additional columns assocaited with each sample. Terra will know to associate the columns with the correct row by your "entity:" column so use the same sample_ids as you did when you originally uploaded the data.

  
 
 
If you did not perform any previous genomic characterization of SARS-CoV-2 sequences on Terra, the assemblies and associated metadata file must first be uploaded to run the Augur workflows (see sub-steps below). 

You have 3 options for uploading assemblies and metadata. These are the same as what you saw in the Titan protocols. 

    1. Upload on Terra (can only do single files).
    2. Upload to Google bucket and link to Terra (can do single files or bulk files/folders).
    3. Upload via 'https://app.terra.bio/#upload' -- This is the easiest option

Here we will show you how to select/upload metadata information to associate with your samples. 

Note: All the variable names and metadata information in the instructions are dummy and are just for demo purpose only.  Please make sure you are using the right variables/names that pertain to your samples.
SECTION: Analysis Inputs/Data uploading
3.1. The first option is to upload a single sample at a time:

Inside your new workspace page, click on the 'Data' panel in the newly created workspace and then click on the 'Files' tab.

 
Once in the 'Files' tab you can either just drag and drop your files into this space or move the mouse over the blue plus sign icon in the bottom right-hand corner and click 'upload'. Upload the sequence files you'll need for this analysis. Each fastq sequence file has to be uploaded individually using this method. 

Once the files are uploaded you will need to bring a table in to associate your files with their corresponding link to their google bucket location.

  
 

Finally, the completed Terra sample file will need to be uploaded to the workspace. On the right-hand side of the workspace 'Data' panel there is a 'Tables' tab. Click the blue plus sign icon on the right edge of the 'Tables' tab. A popup window should appear titled 'Import Table Data'. Select your completed tab-separated sample file for upload and then click the 'Upload' button.

  
If the upload is successful then the sample file should be located under the 'Tables' tab as 'sample (#)' where # is the number of samples in your file. 

  

 The whole process is shown in the video below:
 
You will notice in the video that you need to upload a file with sample metadata and the path to the fasta/fastq files. Once you have uploaded the files into the workspace, you can either copy, paste and save the paths to the metadata/Table file and upload or edit the paths once you have uploaded the metadata (this way is shown in the video)
SECTION: Analysis Inputs/Data uploading
3.2. You will likely have many samples to upload and you can do this by going directly to your Google bucket.

First, go to “DASHBOARD” tab in your workspace and click “Google Bucket” at the bottom right corner of the same page.

  
This will direct you to your “Google Cloud Platform” page for data uploading. 
 
Click “UPLOAD FILES” in the middle of this page to upload single or multiple fastq files. Or you can click “UPLOAD FOLDER” to upload a folder with multiple fastq files stored inside.

  
  
Go back to your Terra account and click “DATA” tab. The successfully uploaded fastq files will show up. If the upload is successful then the files should be located under the 'Files' tab, either inside the folder (as you named) or files

  
Now that the samples are uploaded you will need to return and follow the steps in 3.0 to get your metadata associated with each sample and that should look like below.

  
Here is a short video demonstration:
 

SECTION: Analysis Inputs/Data uploading
3.3. Alternatively, you can upload your files via 'https://app.terra.bio/#upload'. There is no button on Terra to take you to this page, you will need to type this into the search bar. 

Navigate to 'https://app.terra.bio/#upload'

Select the Terra Workspace to which you would like to upload your fastq files. This will be the same workspace created in the previous step.

Click the '+ Create a new collection' link and enter a name for your new collection of fastq files. DO NOT INCLUDE SPACES IN THE COLLECTION NAME, use underscores instead. Spaces will cause an error later in the pipeline.


  
Click the 'Create Collection' button and you will be routed to the data uploader page where you can drag and drop fastq files directly into your browser window to initiate upload.

Drag and drop the fastq files that you would like to upload in to the upload space.

  
Once your files have been successfully uploaded, select "NEXT>" to proceed to the metadata upload page.

  
  
Here is a video of the process:
 

SECTION: Running the Augur Workflows
4.  Running Titan_Augur is a two step process. 
Run Titan_Augur_Prep workflow
Run Titan_Augur_Run workflow
SECTION: Running the Augur Workflows
4.1. Running Titan_Augur_Prep workflow

Navigate to the Augur Prep workflow page:

From 'DASHBOARD' menu -> select 'WORKFLOWS' -> select 'Titan_Augur_Prep' workflow tile

  
This will bring up the Titan_Augur_Prep workflow page:   

  
Select the version of the workflow you would like to run. If no preference pick the latest stable version (in this example v1.4.4). The main and dev versions of the pipeline are under active development and are NOT recommended for users.

Ensure that "Run workflow(s) with inputs defined by data table" is selected, in the Output tab select 'use_defaults, click the "Use call caching" box and then select the root entity type for the sample data you wish to analyze. Don't use the "set" version!   
 

Click "SELECT DATA" and choose the samples you wish to analyze. Note that only 25 samples are shown by default so make sure when you click the check box to select all (as shown in video) that you actually get all the samples you intend to! 

For the inputs to the workflow the top six rows in the input represent variables that have to be provided by the user. Complete the INPUTS section with the appropriate attributes. In our example, for the 'assembly_fastas' variable, for the 'Attribute' text box we set that to 'this.assembly_fasta' to indicate the 'fasta_files' we wish to analyze. The rest of the inputs are as follows:

'collection date' should be 'this.collection_date"
'iso_continent' should be 'this.iso_continent'
'iso_country' should be 'this.iso_country'
'iso_state' should be 'this.iso_state'
'pango_lineage' should be 'this.pango_lineage' --> like assembly fastas this obtained from the previous analysis (Titan or uploaded from your own).

Once you input all the data, it would look similar to the below picture.

  
NOTE: If you named your columns something other than fastas then just type "this." followed by whatever the column name is. We would advise naming your fastas column "fastas" for clarity.

Video showing on how to submit the Titan_Augur_Prep Run:
 

Once the Titan_Augur_Prep workflow runs successfully, you should able to see the sample_metadata.tsv file associated to the sample table data under the TABLE section. This sample_metadata.tsv is selected as one of the inputs to run the Titan_Augur_Run workflow, that is why this step is necessary to check it was generated first.

The video below shows this step:
 

SECTION: Running the Augur Workflows
4.2. Running Titan_Augur_Run workflow

As we did before select the version of the workflow you would like to run (pick the latest stable version, here v1.4.4). The main and dev versions of the pipeline are under active development and are NOT recommended for users.

Ensure that "Run workflow(s) with inputs defined by data table" is selected and click the "Use call caching" box. In contrast to the Titan_Augur_prep workflow, select "augur_prep_sample_set" in the the root entity section, because it will expect to have set of files in order to generate the output for the auspice visualizations. Since our root entity is a sample set when you click "SELECT DATA" click on the set of samples that you just ran Augur_Prep on. 

And in INPUTS section, assembly_fastas attribute is set to "this.augur_prep_samples.assembly_fastas", Remember it will look for set of samples not just sample, so name the attribute to root entity and set it to plural. 

The rest of the inputs are:
Build_name in string format: Here we named it "B.1.147_samples_set", but you can call it whatever you like. 
sample_Metadata.tsvs (Generated by the Augur_Prep in the previous step) as "this.augur_prep_samples.augur_metadata. Again note the plural form!
 

And it should look like this.

  

Once your input form is complete, move on to the OUTPUTS form and select "Use Defaults'. Terra will then populate the OUTPUTS form with all of the default outputs options generated by the workflow. If you forget to do this you won't have easily accessible results! Save these changes by clicking the 'Save' button

Once your INPUTS and OUTPUTS forms are complete, click the 'Save' button on the top right-hand side of the page. The yellow caution icons should disappear and the Run Analysis option should be made available

You are now ready to run the Titan_Augur_Run workflow! Click on the 'Run Analysis' button to the right of the 'Outputs' tab. A popup window should appear titled 'Confirm launch'. If the 'Run Analysis' button is greyed out, you need to save your recent changes by clicking the 'Save' button.

Clicking the 'Launch' button should bring you to the 'Job History' panel where each sample will be queued for the Titan_Augur_Run analysis. The status will change from Queued -> Submitted -> Running

Video showing how to submit the Titan_Augur_Run workflow.
 

SECTION: View and Download the Augur workflow output reports
5. First, verify the sample set has successfully completed by looking at the 'Workflow Status' section in the top left of the 'Job History' panel. The job has completed when the sample set have a status of 'succeeded' with a green checkmark. 

Since Titan_Augur_Run analyzes a set of samples together, the job status would appear this way, it completed analysis successfully (hence the green checkmark).

   
 

Once confirming the job completed, go to the 'DATA' panel and under the 'TABLES' tab click on the sample table that you created and uploaded in step 4. It will be named 'sample (#)' which has the set of samples/entities in your file

  
Now the Terra sample table will have the additional attributes that were added by the workflow when you specified the output names (set to default in this example). You can reduce the number of fields you want to visualize by clicking the "gear" icon in top row on the right. Select only the fields you want to see then click "Done".

  
 
The three files that are used for visualization are:

auspice_input_json - The output generated from the NextClade analysis step of the Augur workflow. This file includes the samples for clade typing and the single sample placed on the tree. Downloading this file won't be saved to your local folders but will be opened in a browser. Make sure you right click on the page and "save as" it to your local directory

combined_assemblies - A concatenation of all of the assemblies that were included in this phylogenetic analysis. For this example it includes all of 20 samples combined into one single filtered fasta file combined_assemblies.

metadata_merged - Every metadata file from each sample that is generated by Titan_Augur_Prep is merged into one single metadata file.

The specific filenames produced by the pipeline can be viewed in the Terra sample report by scrolling to the right (or just only selecting those columns using the "gear" icon). You can download or copy this report by using either the 'Download All Rows' or 'Copy Page To Clipboard' buttons at the top of the table. To download a particular output file, click on the link under the column name a popup window titled 'File Details' should appear.

   
Click the 'Download For < $0.XX' button to save specific files to your local directory for further analyses or visualization.

  
Download all 3 files now to perform the phylogenetic visualization in the next step. 
 
Video showing how to download and save the output files.
SECTION: Auspice Visualization
6. After completing the Augur workflow, the resulting output files contain the comparative metrics necessary to analyze the genetic relatedness between samples and draw phylogenetic inference by visualization with Auspice.

 


You should have downloaded auspice_input_json and metadata merged files from Step 5. We will now use them to generate a phylogenetic visualization using auspice.us. 


  
Data can be uploaded to Auspice by simply drag-and-drop of the auspice_input_json file, which should automatically visualize a phylogenetic tree similar the example below:


  
Hover over the tree and geographical map to see the divergence of the strains, toggle between the options and parameters based on your needs.

Metadata can be added to annotate the tree for assistance with identify local outbreaks or clustered samples by simply drag-and-drop of the metadata_merged file onto the tree.

 
Note: Once you drop the metadata file, all the attributes will show up in the "Color By" section (like in the picture above) where you can visualize your samples by whatever attribute that you have in your metadata file.

Visualization gives few default errors and it would not be a problem, but most importantly it gives the option to color code your samples by metadata variables. 

Since the metadata in this example is random dummy data so results are not accurate, but if you upload your metadata, it would allow you to 'Color By' the metadata variables.

This video shows how to drop and visualize on auspice.
 


SECTION: UShER Visualization
7. It may also be helpful to visualize your results using UCSC's UShER webportal.
 

Start by navigating to the UShER SARS-CoV-2 webportal: https://genome.ucsc.edu/cgi-bin/hgPhyloPlace


 
Upload your concatenated fasta file (combined_assembly from Step 5) by clicking "Choose File" and selecting the combined_assembly file then click "Upload". You can also open the combined_assembly file and then copy and paste the sequence names/IDs. 

UShER is very fast and will place your sequences within a phylogenetic tree of public sequences in GISAID or GenBank within few minutes. After uploading your assembly file pick which phylogenetic tree version you want to use from the drop down menu. These are updated over time as new genomes are added so its a good idea to take note of the version you choose to use. 

UShER outputs subtree results with the 50 sequences (default) most-closely related to your input samples.

You can pick the Phylogenetic tree version if you want to see the from where/how your samples are diverged, UShER would reproduce the results adding your samples to the existing tree nodes. We recommend using the top, most up to date tree. 

Subtrees generated by UShER depend on the divergence among the input samples, if the strains are more divergent, it will produce more subtrees, and if you wish to add more samples, UShER can add them to the existing tree based on sample divergence and mutation. Trees can then be visualize and explore this using the Nextstrain platform.
 


In addition to phylogenetic placement, UShER also provides extra information, including QC metrics, in table format on the UShER results page.

If you wish to view the subtrees, select the options that is available below the table to launch the phylogenetic tree on the Nextstrain platform. Metadata can be added by drag-and-drop similar to with Auspice (Step 6).

For this demo UShER took the concatenated assembly fastas of 20 samples and constructed a phylogenetic tree against all GISAID trees that are being maintained by the UShER and it gave 16 subtrees (See "Subtree Number" column). 

If you hover over the "?" of the column variables/sample rows, a gray dialogue box would appear (like in the picture) showing the info on that attribute. 

 
Colors are represented to show the variations of each attribute between samples.
The lower the score the lighter the color, which means you samples are of high quality when it is green. When the score is high the output would appear in Red. In general the higher the score/red in color in the metrics, suggests that the sample sequence has many errors and sample placements might not be reliable.

As an example:  In the image above the Parsimony score, which explains the number of mutations on the branch, for  Sample_10 is highlighted in red, this says the sample has relatively more mutations added to its branch than the others.


If you want to see the subtrees click the subtree section of the row you are interested in and the link associated to that sample will redirect to the subtree visual rendered on the Nextstrain platform. Nextstrain distinguishes the public samples in Black and your input samples in Red. Subtrees basically shows how your sample is diverged and placed among the public sequences. 

Here is an example subtree:

  
You can also see the subtrees by scrolling down the page and selecting from the individual subtree.

Additional information, such as the likely path of the mutation from root of the tree to your sample, most closely-related sequence in the global tree, and PANGO lineage inference, are provided.


  


SECTION: How to debug a failed run
8. 
1. Failure due to the unavailability of the compute environment. 
The Titan_Augur_Run workflow script requests 78 CPUs in order to perform the phylogenetic reconstruction for visualization in Auspice and Nextstrain. If the analysis fails due to lack of resources, try requesting for more CPUs to Terra. 
  
For more details on how to fix this see step 2 of our Terra Troubleshooting protocol. 

2. Failure for not specifying the input attributes correctly. 

There are two flavors of this error for Titan_Augur_Run:
Singular instead of plural notation
Using sample instead of Sample_Set

First, the singular instead of plural problem:

The Titan_Augur_run workflow looks for concatenated set of fasta files, you need to set the root entity to "name_sample_set" (with name being specific to your dataset) and the input attribute notation of "this.name_samples.assembly_fasta and metadata as "this.name_samples.metadata" rather than using "this.assembly_fasta" and "this.metadata" as we did with the Augur_prep protocol. 

In other words, you select the name_sample_set (singular) in root entity, however for the input use name_samples (plural) notation, since it is looking for all the files as concatenated set.

You will see this error if you didn't use the plural:

 
Second, we will look at the problem of using sample instead of sample set as your root entity type:

  
The specific error message can be found in the draft_augur_tree.log file in the specified Google Cloud storage location. 

 
To avoid this error, ensure the root entity type is set to 'sample_set' and not 'sample', as shown below.

   
3. Error due to spacing in collection name

In this example of a failed run we have an error caused by adding spaces into your collection name. If there is a failure for some other reason you can follow similar steps to understand why your job failed. Here we will be using Illumina paired-end data, but the process of debugging an error is the same if you have ONT data. 

If a run fails you will see this indicated in the job history screen in the "status" column. 

   
To understand why it failed click on the "workflow dashboard" icon in the "links" column. This will take you to a new screen and you can click the arrows next to the "message" to expand the message and see what it says. Here we see there are two errors that direct us to a log file to check. To find out more click on the "execution directory" icon under the links header. This will take you to the google bucket with all the output from the run.

 
Follow the file path to the Google Cloud storage bucket location to the log files referenced in the error messages (in red/orange text in the above photo).  

  
Click on the "Authenticated URL" link that will take you to a text file. 

 
In the text file we can see that there was an error that cause by there being a space between "Bad" and "Sample" in our file path that was created when we made "Bad Sample" rather than "Bad_Sample" as our collection name. 


 
Video of the whole process.  
 
 
Some error message can be difficult to understand. Contact TOAST@cdc.gov for help debugging job failures.