TITLE: SAHDA: Sequence Alignment from Haplotype De Novo Assembly
AUTHORS: Joseph C Blackwell, Megan Mcdonald

[DESCRIPTION]
SAHDA is a Snakemake based workflow tool designed identify and analyse genes/features of interest within small haplotype genomes. The workflow will assemble haploid genomes de novo  from short paired-end reads, create a local BLAST database and query  genes of interest. Where genes of interest are detected, a multiple  alignment is performed for each gene. The tool also allows the input of intermediary files meaning that a combination of both raw sequencing libraries and assembled genomes can be investigated simultaneously.

Primary input: 
Raw paired end sequencing libraries fastq.gz

Optional intermediary inputs/outputs:
Trimmed paired end sequencing libraries fastq.gz
De novo genome assembly fasta
Local BLAST database for each assembly
BLAST search results for each query on each assembly
Retrieved fasta sequences for each query on each assembly

Primary output: 
Multiple alignment fasta file for each query.

GitHub repository: https://github.com/JoeBlackSci/SAHDA

[BEFORE_START]
Assembling genomes de novo takes a long time, so assembled genomes can be input as an intermediary file, allowing the workflow to investigate a combination of raw sequencing reads and assembled genomes without having to repeat any computational analysis. 

If users have access to a HPC where they cannot run SAHDA directly, they may wish to assemble genomes independently and them import them into the workflow.  

[GUIDELINES]
Naming conventions:
When importing intermediary files, they must be named appropriately in order for the workflow to identify them as files to incorporate into the workflow. 

For example, these are the file and directory names required for a config.yml file containing the following specifications.

Note: importing an assembled genome requires the genome to be in fasta format, named 'scaffolds.fasta' and placed inside a directory named '<sample_name>_assembled', that is in turn placed inside the '02_assemble'  Directory (see below).

 

 


[STEPS]
SECTION: Overview
1. The following is an overview of the workflow. When run, the workflow will automatically take inputs and perform subsequent steps. Each step can act as an input for files and generates an intermediary output that can be extracted from the workflow. The software used in each step is linked and the inputs and outputs of each step is listed below.

1. Trimming of raw fastq files [trimmomatic] 
    Input:         Raw paired end sequencing files in fastq.gz format
    Output:     Trimmed and QC controlled fastq.gz files

2. De Novo assembly [SPAdes]
    Input:       Trimmed paired end and unpaired sequencing fastq.gz files
    Output:    Assembled genome scaffold.fasta file

3. Create local BLAST databases for each de novo assembly [BLAST+]
    Input:       Assembled genome scaffold.fasta file
    Output:    Indexed local BLAST database for each 

4. Query each assembly BLAST database with gene of interest [BLAST+]
    Inputs:     Local BLAST database and sequences to query in .fasta format
    Output:    BLAST search results

5. Extract sequence from the database [BLASTtoGFF] 
    Input:       BLAST search results and assembled genome scaffold.fasta
    Output:    Separate retrieved fasta files for the best BLAST hit

6. Combine fasta files based on query sequence
    Input:       Separate fasta files
    Output:    Combined fasta files 

7. Perform multiple alignment for each identified gene [MAFFT] 
    Input:       Combined fasta files 
    Output:    Multiple DNA alignment fasta file
SECTION: Installation
2. Download SAHDA via either of: 

a. zip from the git repository: https://github.com/JoeBlackSci/SAHDA

  
b. git clone command on the commandline

 

SECTION: Installation
4. Install Prerequisite Programs

1. Install Miniconda 3 using the miniconda3 installation instructions. 

This will install the latest version of the Conda environment management software and it's minimal dependencies.

2. Install Snakemake using Conda.

Snakemake is a python based workflow management language for reproducible and scalable data analysis. It is recommended to recreate the snakemake environment used to develop this workflow.

To do so, navigate using the commandline, to the top level of the workflow directory (containing the README.md file, see above):
 

And recreate the snakemake environment from the environment.yml "env_snakemake_mgw.yml" file in the config folder using:
 

SECTION: Test Installation
5. Before running your own data, it is recommended to run the included example workflow. This will test that the workflow is working correctly, generate the intermediary file structures, provide example files at each step and download software required for the workflow via conda.

Navigate using the commandline, to the top level of the workflow directory (containing the README.md file, see above)
 
Activate the snakemake conda environment
 
Run the example workflow
 

SECTION: Installation
3. 
After unpacking or cloning the repository, you should see the following file structure.
 


SECTION: Test Installation
6. Running the example workflow successfully will result in the following file structure and results files:
 

SECTION: Input Data
7. Paired end reads

Paired end files should be added to the `resources/reads` directory, and the sample name added to the config file described later.

This workflow will accept paired-end short read sequences with filenames of the following formats.
 
Note: The workflow currently only supports samples spread over two lanes. i.e. only L001 and L002 are currently supported.

```
SECTION: Input Data
8. Adapters

The adapter file, in `.fasta` format, should be added to the `resources/adapters` file and the full filename specified in the `config/config.yml` file described later.
SECTION: Input Data
9. Target gene database

Query genes should be added as individual `.fasta` files to the `resources/query` directory.
SECTION: Configuring the Workflow
10. Configuration

The workflow configuration file `config/config.yml` can be edited as a text file to specify parameters for the workflow. Including samples, query genes and trimming options. See the config.yml file for all options.
SECTION: Configuring the Workflow
10.1. Sample Names
Sample names should be included without lane, direction or file extension. For example: 
The paired end library EXAMPLE1_R1.fastq.gz and EXAMPLE1_R2.fastq.gz should be included as EXAMPLE1.
The paired end library EXAMPLE2_L001_R1.fastq.gz, EXAMPLE2_L001_R2.fastq.gz,  EXAMPLE2_L002_R1.fastq.gz and EXAMPLE2_L002_R2.fastq.gz should be included as EXAMPLE2.
 

note: if your filename includes a lane number but is not split over multiple lanes, then the lane number should be edited out or included in the sample name in the config file.

If you are using intermediary files as inputs, ensure that they are placed in the correct directory and follow the example naming conventions.
SECTION: Configuring the Workflow
10.2. Query Database
The names of individual query sequence files should be included without file extension. For example:
The EXAMPLE-QUERY.fasta query should be included as EXAMPLE-QUERY.
 

SECTION: Configuring the Workflow
10.3. Adapter file
The name of the adapter file including file extension.
 

SECTION: Running the Workflow
11. Activate the Environment 

To run the workflow, navigate using the commandline to the top level directory containing the README.md file.
 

Activate the workflow's conda environment.
 

SECTION: Running the Workflow
12. Executing the Default Workflow

To execute the default workflow run the following command
 
Note this will use the maximum cores available on your local machine. To specify the number of cores use --cores N where N is the maximum number of cores to be assigned.

Snakemake will automatically determine which jobs need to be run to  generate all target files. Meaning that output files that are already present will not be regenerated.

If you have input intermediary files such as an already assembled genome, then SAHDA will also run on these files so long as the sample is named in the config. 

After running this command, your multiple alignment files will be found in 'results/08_align/'
SECTION: Optional Parameters
13. Parameters For a full list of snakemake parameters see the snakemake documentation.

[--use-conda] Required This workflow should always be run with with --use-conda  flag to enable each rule to download and utilise a specific conda  environment. This ensures a reproducible workflow, but also ensures that  required software will not conflict.

[-j --jobs --cores] Required Specify the maximum number of CPU cores to use in parallel. Specify all to use all cores available.

[-n --dryrun] Dry run the Snakefile with -n prior to performing an analysis to check for potential errors.

[-p --printshellcmds] Print the shell commands to be executed to the terminal.

[-r --reason] Print snakemake's reason for executing each job.

[-R --forcerun] List the rules that should be re-run.  Snakemake does not re-perform analysis if new files are added prior to a  step that combines outputs from previous steps. For example if new data  has been introduced prior to a multiple alignment. In these instances -R mafft-align would need to be specified. A full list of snakemake rules can be seen in the Rules List.

--list-input-changes gives a list of output files that are affected by the inclusion of new data. This can be used to automatically trigger a --forceun with the following code.