Bioinformatics on the cloud. Aligning RNAseq with STAR on Google... | by Hylke C. Donker | Towards Data Science

The price of analysing a person’s DNA, as well as its transcribed instructions (RNA), are rapidly getting cheaper. As a result, uptake of sequencing technology is growing, not only in scientific labs, but also in routine healthcare. But the data that is generated for a single individual can easily exceed several GBs in size. So this truly is a big data problem! Since the cloud is specifically designed to address problems of large scale, we can leverage its tools and infrastructure to boost our bioinformatics analyses. By running your bioinformatics tools on the cloud, you can do the analysis in a fraction of the time.
This tutorial will walk you through the steps of aligning human RNA sequencing data (RNAseq, for short) on Google Cloud Platform (GCP).
Objectives:
Store genomic data on the cloud.
Align RNA sequencing data using STAR.
Sequence machines, that read nucleotide bases off a strand, can only read small portions of a fragment. The machine therefore generates a large collection of small chunks of bases, called reads. Usually, reads come in two flavours: left and right reads. These reads, generated by paired-end sequencing machines, are the two sides of a fragment from where the machine started reading the bases. After finishing the measurement, the sequencing machine typically generates two files (one for each flavour) containing all these chunks of bases with corresponding quality estimates in FASTQ format. All these small pieces of the puzzle, that are now in FASTQ format, have to be put back together. This is the process of alignment against a reference genome, where the reads are assigned to a location on the genome. When successful, this will result in a SAM file indicating the genomic location of the reads. Our task for today is to carry out the alignment of bases that originate from RNA material. Or put differently, combine and transform the FASTQ files to SAM files.
Open the Google Cloud Platform dashboard and click on the cloud shell icon (Fig. 2) in the top right corner to launch the terminal.
We will make use of dsub to distribute our bioinformatics workload on Google Cloud Platform. To install dsub, execute:
sudo pip3 install dsub
in your cloud shell, or on your local machine, if you have installed and configured gcloud locally.
Next, choose a name for storing your data, e.g., gs://rna-seq-tutorial, and make a bucket (mb):
export BUCKET=gs://rna-seq-tutorialexport REFDATA=${BUCKET}/refdataexport INPUT=${BUCKET}/inputexport GENOME_INDEX=${BUCKET}/genome_indexgsutil mb -l europe-west4 ${BUCKET}
And download the annotation file to the newly created bucket:
wget -O - \    ftp://ftp.ebi.ac.uk/pub/databases/gencode/Gencode_human/release_36/gencode.v36.chr_patch_hapl_scaff.annotation.gff3.gz \    | zcat \    | gsutil cp - ${REFDATA}/gencode.v36.chr_patch_hapl_scaff.annotation.gff3
This command downloads the file (wget), unpacks it (zcat) and copies the data to the bucket (gsutil cp). Next, copy the FASTQ files (the RNAseq data in raw format) that are to be aligned to the bucket. If you are not using your own files, use the following example files [1]:
wget  http://genomedata.org/rnaseq-tutorial/practical.tartar -xf practical.tar# Sample 1.gsutil cp \    hcc1395_normal_rep1_r1.fastq.gz \    ${INPUT}/sample1_R1.fastq.gzgsutil cp\   hcc1395_normal_rep1_r2.fastq.gz \   ${INPUT}/sample1_R2.fastq.gz# Sample 2.gsutil cp \    hcc1395_tumor_rep1_r1.fastq.gz \    ${INPUT}/sample2_R1.fastq.gzgsutil cp\   hcc1395_tumor_rep1_r2.fastq.gz \   ${INPUT}/sample2_R2.fastq.gz
And finally, enable the Cloud Life Sciences API, to use dsub.
To align the RNA transcripts to the reference genome, we will make use of STAR [2]. Alignment with STAR is a two-step process:
Generate a genome index using genome reference information.Align sequencing data using the genome index.
Generate a genome index using genome reference information.
Align sequencing data using the genome index.
STAR uses both the reference genome and the annotation file to generate the index files. Create a bash script called step1.sh with the following content:
#!/bin/bashSTAR --runThreadN 8 \    --runMode genomeGenerate \    --sjdbOverhang 100 \    --genomeFastaFiles ${FASTA_FILE} \    --sjdbGTFfile ${GTF_FILE} \    --genomeDir ${GENOME_INDEX}
This will use 8 cores to generate the index files, and write them to $GENOME_INDEX (the location will be specified below). We passed the reference genome using the --genomeFastaFiles flag, while the gene annotation file can be specified with --sjdbGTFfile. For more details on the command line argument, please refer to the STAR manual. Next, use dsub to execute the workload on a worker with 8 virtual cores (--min-cores 8) and 48 GBs of memory (--min-ram 48), since STAR requires loads of memory for this step:
dsub \    --provider google-cls-v2 \    --project <my-project> \    --location europe-west4 \    --zones europe-west4-a \    --preemptible \    --min-ram 48 \    --min-cores 8 \    --logging "${BUCKET}/logging/" \    --input FASTA_FILE=gs://gcp-public-data--broad-references/hg38/v0/GRCh38.primary_assembly.genome.fa \    --input GTF_FILE=${REFDATA}/gencode.v36.chr_patch_hapl_scaff.annotation.gff3 \    --output-recursive GENOME_INDEX=${GENOME_INDEX} \    --image registry.gitlab.com/hylkedonker/rna-seq \    --script step1.sh
Decide if you want to go for the cheaper pre-emptible compute (--preemptible) at the risk of job termination, and replace <my-project> with your Google Cloud Platform project ID. You can adjust the location as desired, but you are advised to adjust your bucket location accordingly. Notice that there was no need to spin up a virtual machine, and install our bioinformatics tools. Instead we used a Docker image (--image registry.gitlab.com/hylkedonker/rna-seq) that contains everything we need, and dsub takes care of the rest. While our workload is being executed on the cloud, we can prep for the next step.
To do the actual alignment, create a file step2.sh with the following contents:
#!/bin/bashSTAR \    --runThreadN 4 \    --readFilesCommand zcat \    --genomeDir ${GENOME_INDEX} \    --readFilesIn ${R1} ${R2} \    --outFileNamePrefix "${OUTPUT}/"
This script invokes STAR to do the actual alignment on four cores (--runThreadN 4) using the index that was generated in the previous step (--genomeDir ${GENOME_INDEX}). Notice that the alignment of each sample is (computationally) a self contained problem. This means all samples can be aligned in parallel! Executing arrays of jobs is easy with dsub. Simply create a tab-separated file (a .tsv file) with the arguments to be passed to dsub. For this tutorial, create a file called job2.tsv with the following contents:
--input R1 --input R2 --output-recursive OUTPUTgs://rna-seq-tutorial/input/sample1_R1.fastq.gz gs://rna-seq-tutorial/input/sample1_R2.fastq.gz gs://rna-seq-tutorial/output/step2/sample1gs://rna-seq-tutorial/input/sample2_R1.fastq.gz gs://rna-seq-tutorial/input/sample2_R2.fastq.gz gs://rna-seq-tutorial/output/step2/sample2
Replacing <my-bucket> with the appropriate value. Also, notice that the columns are tab separated (no white spaces). Before executing the alignment workloads, make sure that STAR finished generating the genome index. The status can be queried using dstat. After verifying the job was successfully carried out, submit the alignment workloads to the cloud using:
dsub \    --provider google-cls-v2 \    --project <my-project> \    --location europe-west4 \    --zones europe-west4-a \    --preemptible \    --min-ram 32 \    --min-cores 4 \    --logging "${BUCKET}/logging/" \    --input-recursive GENOME_INDEX=${GENOME_INDEX} \    --image registry.gitlab.com/hylkedonker/rna-seq \    --tasks ./job2.tsv \    --script step2.sh
Here, we used --tasks ./job2.tsv to refer to input and output arguments, one line per sample. Finally, wait for the workers to finish (use dstat to poll the status) and don't forget to remove your resources after finishing your computation to prevent additional storage costs.
gsutil rm -r ${BUCKET}
That’s it, you have just aligned your first RNAseq data on the cloud using STAR! Notice how easy it is to scale up our analysis: aligning 1000 instead of 2 samples is simply a matter of extending the tab-separated file.
Of course, alignment by itself is usually a pre-processing step required for downstream analysis. In part II, we will continue our cloud adventure by computing gene expression from our aligned RNAseq.
Drop a comment if you have helpful STAR tips or interesting cloud tricks to share, or any improvements to the code.
[1]: Malachi Griffith*, Jason R. Walker, Nicholas C. Spies, Benjamin J. Ainscough, Obi L. Griffith*. 2015. Informatics for RNA-seq: A web resource for analysis on the cloud. PLoS Comp Biol. 11(8):e1004393.
[2]: Dobin, Alexander, et al. “STAR: ultrafast universal RNA-seq aligner.” Bioinformatics 29.1 (2013): 15–21.