TITLE: ICGRC Portal Tripal Data Generation and Setup
AUTHORS: Locedie Mansueto, ramil.mauleon, Tobias Kretzschmar, Graham King

[DESCRIPTION]
    The data provided by the International Cannabis Genomics Research Consortium (ICGRC, https://icgrc.info) web portal consists of published results from past analyses, and results generated using the steps described in this protocol. We used public datasets, and open software  commonly used for the specific tasks, suggested from best practices, or from other Tripal sites.  The sections in this protocol can be grouped into Data Generation (steps 1-8), Setup Tripal and Modules (steps 9-15), Multi-omics API Integration (step 16,17), and Setup of non-Tripal pages (step 18). 

[STEPS]
SECTION: Prepare RNA-Seq Sequences
1. Next Generation Sequencing (NGS) RNA-Seq sequences are downloaded from NCBI SRA, and trimmed to remove adapters. Three sets of RNA-Seqs are prepared for: 1) Purple Kush gene prediction, 2) Finola gene prediction, 3) transcript assembly, expression level, and variant discovery for trichomes from 21 samples
SECTION: Gene functional annotation
3. Functional annotation by sequence homology using mmseqs2, is a fast sequence alignment alternative to BLAST. 
 

 

SECTION: Transcript assembly and alignments
4. Four sources of Cannabis transcripts use are: 1) cs10 mRNA from NCBI RefSeq, 2) transcript assembled by publications,  3) gene prediction using cultivar-specific RNA-Seq sequences in step 2., 4) assembled transcripts using trichome RNA-Seq of 21 cultivars from multiple sources. The transcripts in this step are loaded and visualized in the JBrowse genome browser.
  

SECTION: Gene expression
5. Gene expression quantification uses Salmon in mapping-based mode https://salmon.readthedocs.io/en/latest/salmon.html#preparing-transcriptome-indices-mapping-based-mode
We quantify the expression of the cs10 RefSeq mRNA.

 

 

 

SECTION: Genotype
6. SNP genotype dataset generation is described in details in separate protocols 
 

SECTION: Synteny
8. MCScanX was used to detect synteny blocks using the genes for cs10 and pkv5 https://github.com/wyp1125/MCScanX

 

 

SECTION: Biopathways
7. The biopathways module visualize the expression level or differential expression of transcripts/proteins over biological pathways. It is a web-based implemantation of MapMan
https://github.com/usadellab/usadellab.github.io/tree/master/MapManJS
SECTION: Gene models prediction
2. Generation of gff and fasta files. Unless gene models and annotation are available in RefSeq for a genome assembly like cs10, local gene prediction and annotation are preformed. Cultivar-specific RNA-Seq datasets are used whenever available.

 


 


SECTION: Gene models prediction
2.1. Generate metadata csv listing the RNA-Seq sequences to use. Examples for Purple Kush and Finola
    
Prepare the STAR and olego genome  index. FINDER wrongly parse a dot/period (.) in the contig IDs.  As most contig IDs from NCBI ReqSeq/GeneBank have dots, rename the IDs by replacing dots with underscore
 

SECTION: Gene models prediction
2.2. Run FINDER. 
 

The final outputs are generated in the directory ${FINDER_WORKDIR}/${FINDER_OUTDIR}. It is observed that multiple gene models are overlapping and fragmented in a region. These models are clustered in the next step using the output gtf file.

SECTION: Gene models prediction
2.3. Cluster gene models using gffread. 
It is observed that multiple gene models are overlapping and fragmented in certain regions. These models are clustered in this step from the FINDER output gtf file.


 

 


Merged gene models are listed in ${FINDER_GTF}.dupinfo. The clustered gene models in CLUSTERED_GFF is in gff format. The remaining files are sequences for CDS, exons and proteins in fasta format.
SECTION: Gene models prediction
2.4. Revert the contig names in CLUSTERED_GFF back to the original names used in REF_FASTA.  As an option,  the clustered output gene model IDs (gene ids, cds id, protein ids, exon ids) may be renamed using your own nomenclature. 
SECTION: Gene functional annotation
3.1. Download the latest Uniprot viridiplantae proteins sequence and generate mmseqs2 database
 

SECTION: Gene functional annotation
3.2. Get best hit alignment between predicted proteins with Uniprot plants

 

Assign the homologs functional annotation and scores (target ID, e-value) to the Description and/or Note attribute in the GFF.
SECTION: Gene functional annotation
3.3. Functional annotation by Interpro. Submit CLUSTERED_PEP to InterproScan to any of the public Galaxy servers (EU,US or AU). Include Gene Onlogy ID (GO_ID) assignment, and set outputs to xml and tsv formats.
SECTION: Transcript assembly and alignments
4.1. (Optional) Assemble RNA-Seq sequences into transcripts. rnaspades is used for its speed.

 


 


The output assembled transcript sequences are in file $TRANSCRIPT_SPADES_FASTA

SECTION: Transcript assembly and alignments
4.2. Align transcripts to genome into bam file. The results are visualized in the JBrowse genome browser.

 


 


 


SECTION: Gene expression
5.1. Create decoy-aware transcriptome file https://combine-lab.github.io/alevin-tutorial/2019/selective-alignment/. This is generated by appending the genome sequences to the transcripts sequences, and list the decoys IDs. The decoys are the genome sequences. 
 

SECTION: Gene expression
5.2. Create salmon index file
 

SECTION: Gene expression
5.3. Quantify each replicate
 

The quantified expression is in file $SALMON_DIR/*/quant.sf

Estimate abundance
 

SECTION: Biopathways
7.1. Download MapMan mapping files, curated pathways, and protein sequences for Arabidopsis thaliana, Eucalyptus grandis (eucalypthus), and Solanum lycopersicum (tomato) 
 
Arabidopsis thaliana map
X4 Araport11
https://mapman.gabipd.org/mapmanstore?p_p_id=MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5&p_p_lifecycle=0&p_p_state=normal&p_p_mode=view&p_p_col_id=column-1&p_p_col_pos=1&p_p_col_count=2&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_Show=DownloadMapping&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_RessourceId=411&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_Download=TextMapping


 


Eucalyptus grandis map
Egrandis_201 
https://mapman.gabipd.org/mapmanstore?p_p_id=MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5&p_p_lifecycle=0&p_p_state=normal&p_p_mode=view&p_p_col_id=column-1&p_p_col_pos=1&p_p_col_count=2&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_Show=DownloadMapping&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_RessourceId=336&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_Download=TextMapping


 

Solanum lycopersicum map
ITAG2.3
https://mapman.gabipd.org/mapmanstore?p_p_id=MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5&p_p_lifecycle=0&p_p_state=normal&p_p_mode=view&p_p_col_id=column-1&p_p_col_pos=1&p_p_col_count=2&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_Show=DownloadMapping&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_RessourceId=310&_MapManDataDownload_WAR_MapManDataDownloadportlet_INSTANCE_4Yx5_Download=TextMapping

 

 

 

SECTION: Tripal modules and data loaders
9. Install and enable the following Tripal modules, then load the different datasets corresponding to each module. The step-by-step guide to loading are described in each modules documentations in the provided links. The purpose of this section is to summarize the datafiles and format needed for each module,  how to generate them using the results from the previous sections, or where to find them if publicly available. 
Datasets generated from the previous sections and needed in the next steps are compiled in https://icgrc.info/downloads  

SECTION: Biopathways
7.2. Align protein of cs10 vs the three plants' proteins using mmseq reciprocal best hit
 

SECTION: Biopathways
7.3. Revise MapMan maps to use the cannabis genes using the reciprocal best hit results. 
   
  
  



SECTION: Multiomics API Use-cases
17. These scripts demonstrate the use of the API to query various datasets and do some plotting and analysis. The  web-service API documentation is available at https://icgrc.info/api_doc

The Jupyter notebook ipynb uses the client modules defined in omics_api.py, using functions defined in omics_api_utils.py and the scripts in api-modules.tgz. Static export pages of the notebook on different use cases are available at https://snp.icgrc.info/static/icgrc_omics_demo.html. Module updates are also announced/available at this page between protocol version updates.

        

SECTION: Tripal modules and data loaders
11. Install Mainlab Chado Search module
https://gitlab.com/mainlabwsu/chado_search

This module uses the genome features and annotations analyses loaded in steps 2 and 3, to create the Materialize View chado_search_gene_search. 
The default query definition is modified in the Materialized Views page to account for i) the nomenclature of the different annotations and references, ii) to include associated mRNA and protein annotations in gene accession and keyword search, and iii) include transitive closure when using Gene Ontology terms in keyword search. The modified SQL definition is in   

SECTION: Tripal modules and data loaders
12. Install Expression module
https://github.com/tripal/tripal_analysis_expression
SECTION: Tripal modules and data loaders
13. Install Phenotype module
https://analyzedphenotypes.readthedocs.io/en/latest/index.html
This module heavily relies on ontology terms to define phenotypes. There are two ways of loading ontologies supported by this module:
1. use published Bioontologies
2. use customized/specialized ontology/controlled terms
In addition, there are already loaded default ontologies in Tripal for method NCIT and unit UO
SECTION: Tripal modules and data loaders
14. Map viewer
https://gitlab.com/mainlabwsu/tripal_map
Maps consists of a map with and set of features (genetic markers, QTLs, physical markers)
SECTION: Embedded non-Tripal pages
18. These tools are not part of Tripal, but are instantiated and embedded as Tripal pages in the ICGRC web portal. 
SECTION: Multiomics queries
16. The multiomics data integration is implemented as a web-service API. The documentation in https://icgrc.info/api_doc describes the queries and applicable constraints.  It is uses a Drupal module to query the various materialized views generated by the Tripal modules.

SECTION: Tripal modules and data loaders
10. Create and load the basic information for a new genome. 
SECTION: Tripal modules and data loaders
10.1. Create an organism https://tripal.readthedocs.io/en/latest/user_guide/example_genomics/organisms.html
In Chado, the organism_id is used to define a genome assembly. Multiple assemblies or versions for a species can be assigned as different "Organisms" in Tripal. 
Create Genome Assembly (Analysis), and load reference genome FASTA file, associated to Analysis and Organism
SECTION: Tripal modules and data loaders
10.3. Define the gene modelling analysis and load the GFF gene models generated in step 3.2, or downloaded from sources like NCBI RefSeq. 
https://tripal-devseed.readthedocs.io/en/latest/loading_GFF.html
SECTION: Tripal modules and data loaders
10.2. Define the Gene Annotation (Analysis) used to generate the gene sequences. 
Load and publish mRNA and protein sequences FASTA files generated from step 2.5 or downloaded from sources like NCBI RefSeq, associated to Analysis and Organism
https://tripal-devseed.readthedocs.io/en/latest/loading_FASTA.html

 
Download cs10 sequences and GFF files
 

 

 

Steps 10.2 and 10.3 are interchangeable, provided the sequence IDs in the FASTA and GFF files should match.  
SECTION: Tripal modules and data loaders
10.4. Load additional annotations generated from different analysis. In all analyses always generate the XML output format when available, since it is the most comprehensive and required by the Tripal loaders. As a rule, always read the loaders documentation first before running an analysis so that the accepted formats are generated. 
 
Load the InterproScan result in step 3.3
https://tripal-devseed.readthedocs.io/en/latest/loading_IPS.html
and additional BLAST results performed
https://tripal-devseed.readthedocs.io/en/latest/loading_BLAST.html
SECTION: Tripal modules and data loaders
15. Install Synteny module
https://github.com/tripal/tripal_synview


SECTION: Tripal modules and data loaders
12.1. Create the biosample TSV file, or download them from NCBI BioSample as XML is available. 
For TSV, the columns should be as shown in this example
https://github.com/tripal/tripal_analysis_expression/blob/master/example_files/exampleTSV.tsv

organism should match the organism name defined in 10.1 
sample_name should match previously loaded Biomaterial names if available
temp, tissue, treatment should match previously loaded Ontology terms if available

Load and publish BioSample
SECTION: Tripal modules and data loaders
12.2. Load expression data generated in step 5.3, or from public expression repository like NCBI GEO.

The expression data can take a TSV format with sample name as column header, feature name (gene, mRNA, protein name) as row header, and expression value at each matrix element. Samples (12.1) and features (10.2) should be loaded first as described, and the names should match. 
A sample expression matrix available in
https://github.com/tripal/tripal_analysis_expression/blob/master/example_files/exampleMatrix.tsv

This module creates the expression_feature_all materialized view
SECTION: Tripal modules and data loaders
13.1. Download OBO files from published sources
For popular crops, use crop-specific terms from http://www.cropontology.org
In addition, use generic plant and trait terms from TO https://obofoundry.org/ontology/to and PATO https://obofoundry.org/ontology/pato.
For metabolites, use ChEBI https://purl.obolibrary.org/obo/chebi.obo

Load all OBO files necessary 
https://tripal.readthedocs.io/en/latest/user_guide/example_genomics/controlled_vocabs.html
SECTION: Tripal modules and data loaders
13.2. Setup Trait Ontologies using any of the loaded Ontologies
https://analyzedphenotypes.readthedocs.io/en/latest/admin_guide/setup.html
Check Allow new terms to be added to the Controlled Vocabulary during upload for flexibility
SECTION: Tripal modules and data loaders
13.3. Upload phenotype data in TSV format
https://analyzedphenotypes.readthedocs.io/en/latest/user_guide/loading.html 

In the tsv, the Trait Name, Method Name, Unit columns should match the ontology terms when available
Germplasm Accession, Germplasm Name should match preloaded Stocks when available

Stocks may also be preloaded in bulk
https://tripal.readthedocs.io/en/latest/user_guide/bulk_loader.html
or Stock https://icgrc.info/bio_data/add/37, 
or Germplasm https://icgrc.info/bio_data/add/21

This module creates the mview_phenotype materialized view
SECTION: Tripal modules and data loaders
14.1. Create map
Map https://icgrc.info/bio_data/add/16
SECTION: Tripal modules and data loaders
14.2. Load map features
https://icgrc.info/admin/tripal/loaders/cmap_loader
SECTION: Synteny
8.1. Align two reference gene models using blastp
 

SECTION: Synteny
8.2. From the GFF file,  generate TSV file with columns contig,gene_id,start,stop then rename contigs into annotation code plus incremental number, ie. cs1, cs2 .. csN, pk1,pk2,..pkN. Also generate a TSV mapping file of the codes and original contig names.  Concatenate the files for the pair.

SECTION: Synteny
8.3. Create a directory and move the input files. Run MCScanX
 

File ${PAIR_NAME}/${PAIR_NAME}.collinearity is generated
SECTION: Tripal modules and data loaders
15.1. Copy the files generated from steps 8.2 and 8.3 to the tripal_synview directory
 

SECTION: Tripal modules and data loaders
15.2. Load the generated file ${PAIR}.block.4tripal.txt to Tripal and use in defining a new Synteny page
SECTION: Multiomics queries
16.3. The general structure is to query triples of (DATATYPE-PROPERTY,BIOMATERIAL,VALUE). The queries as shown below don't include the WHERE clauses, which are dynamically generated from the arguments of the API call.

Query expression values, with datatype EXP
 
Query phenotype values, with datatype PHEN
 
Query biosample attributes/metadata, with datatype ID or PROP
 


Query SNPs, with datatype SNP
 
In the API module, call bcftools to generate the three files pos_nnnnnn.txt, samples_nnnnnn.txt and call_nnnnnn.txt using the $REGION and $SELECTED_DATASET parameters, and nnnnnn is randomly generated per request. $REGION are the genomic regions from the API argument, or from genomic regions of genes returned by keyword search in <gene_function_search>.  $SELECTED_DATASET is the vcf file to use based on the reference and dataset from the API arguments.

 
Reading the generated files, create three temporary database tables pos_nnnnnn for the features, col_nnnnnn for the samples, and call_nnnnnn for the call at a feature and sample. Then the following query returns the triples of (POSITION, BIOMATERIAL,CALL).
 
chado.mview_srr_prjn_samn_name is a utility materialize view that maps the SRR,SAMN,PROJ IDs and names of the samples.
SECTION: Multiomics queries
16.4. The queries from the different datasets are merged using UNION of (PROPERTY,BIOMATERIAL,VALUE). For sorting and grouping purposes, the DATATYPE column with  possible values [ID, PROP, PHEN, EXP or SNP] is prepended for each PROPERTY  to identify the row datatype.  
First define the colpivot function (https://github.com/hnsl/colpivot) inside Postgres
 
Pivot into table with row headers PROPERTY, column headers BIOMATERIAL, and element values VALUE
 

SECTION: Multiomics queries
16.1. The URL queries are processed using Drupal simple_slim_api module https://www.drupal.org/project/simple_slim_api. To instantiate, install and enable the simple_slim_api module, then replace simple_slim_api.module with this customized script   , and utility script  . Set the bcftools path and vcf files location in read_snps.php

In the module, the URL arguments are converted into SQL queries on the materialized views created by the modules in section <Tripal modules and data loaders>.  
SECTION: Embedded non-Tripal pages
18.1. Instantiate JBrowse v1 following the manual at https://jbrowse.org/docs/tutorial_classic.html
 
Load the genome assembly, GFF gene model tracks (from step 2, or 10.2),  transcript BAM alignment and count bigwig tracks (step 4)
 
Load the GFF gene models generated in step 3.2, or downloaded from NCBI RefSeq in step 10.2
 
Add the BAM file TRANSCRIPT_BAM from step 4.2 to visualize as track.

Install and configure the Tripal-JBrowse Integration module from https://tripal-jbrowse.readthedocs.io/
SECTION: Embedded non-Tripal pages
18.4. MapManJS
 
Place the mapping files from step 7.3 into the MapManJS MappingFiles directory, and reference them in ultramicro.html mappings[] options.

 

The customized ultramicro.html   

SECTION: Embedded non-Tripal pages
18.3. Hemp-Seek 
 
Load VCF files generated in step 6
SECTION: Multiomics queries
16.2. These materialize views were generated by the <Expression> and <Phenotype> modules
 

SECTION: Prepare RNA-Seq Sequences
1.1. Download and extract sequences from NCBI-SRA. The SRR fastq sequences can be paired-end (in SRRLIST_PE) or single end (in SRRLIST_SE)
          

 

 

SECTION: Prepare RNA-Seq Sequences
1.2. Trim using trimmomatic. The paired-end and single-end fastq are iterated separately since they require different command arguments 

 

 

SECTION: Prepare RNA-Seq Sequences
1.3. (Optional) Concatenate multi-fastq samples. BioSample SAMN with multiple SRR fastq sequences
  

 

SECTION: Tripal modules and data loaders
9.1. Install Tripal v3 docker
 

SECTION: Embedded non-Tripal pages
18.2. The following modules use the Drupal Iframe module to embed non-Tripal modules into Tripal pages. 
Download from https://www.drupal.org/node/297891 into the modules directory, extract and enable the module. Once each tools are setup in their respective directories, reference their URL address in the Iframe definition pages. 
SECTION: Transcript assembly and alignments
4.3. Find open reading frames (ORF) of assembled transcripts to within the reference genomes. Transdecoder is used following https://github.com/TransDecoder/TransDecoder/wiki
The transcripts are aligned to genome using GMAP https://github.com/juliangehring/GMAP-GSNAP

Build the GMAP genome database


 

 

Map assembled transcripts to find ORFs
 

Search for long ORFs and peptides from assembled transcripts
 


Longest ORF peptides are aligned to UniProt plants, using mmseqs similar to step 3.2. Best hit results is in file in LONGESTORF_MMSEQ_BESTHIT
 

Run TransDecoder ORF prediction, generating output ${TRANSCIPT_FASTA}.transdecoder.gff3
 


Generate ORF with genome coordinates
 


Cluster overlapping gene loci, and generate CDS, exon and protein sequences

 

 


SECTION: Gene expression
5.4. (Optional) Generate bigwig files to display abundance level as JBrowse track using https://www.encodeproject.org/software/bedgraphtobigwig plugin. 

 

 

Edit the filenames in   using the outputs from steps 5.3 and 4.3, and run the python script. These chromosome length files are also needed