Next-Generation Sequencing Data Analysis With PySpark | by Aneesh Panoli | Towards Data Science

With DNA, you have to be able to tell which genes are turned on or off. Current DNA sequencing cannot do that. The next generation of DNA sequencing needs to be able to do this. If somebody invents this, then we can start to very precisely identify cures for diseases. — Elon Musk
The rapid growth of Next Generation Sequencing technologies such as single-cell RNA sequencing(scRNA-seq) demands efficient parallel processing and analysis of big data. Hadoop and Spark are the goto opensource frameworks for storing and processing massive datasets. The most significant advantage of Spark is its iterative analytics capability combined with in-memory computing architecture. Calling .cache() on a resilient distributed dataset (RDD) effectively saves it in memory and makes it instantly available for computation, thus the subsequent filter, map, and reduce tasks become instantaneous. Spark has its query language known as Spark SQL, and its MLlib library is highly desirable for machine learning tasks.
The scRNA-seq can do what Musk pointed out — figuring out which genes are turned on/off at cellular resolution. But it’s not the silver bullet — because the regulation of gene expression doesn’t stop at the transcription level. The mRNA or the messenger RNA — a proxy of protein-coding regions of the gene — also contains sequences that determine the level of ribosomal protein synthesis. The expressed protein can also undergo a series of post-translational modifications that can activate or repress it.
Over the years, I have come to love Google Colab for all my quick and dirty project prototyping. With Colab, you can skip all the initial steps required to get your project off the ground. No need to set up a virtual environment, dependencies, etc. Also, Colab comes with free GPU/TPU for all your machine learning needs.
First, go to Colab following the link above and fire up a new Python 3 notebook. PySpark requires Java (Java 1.8) and Scala, so next, we will install them.
Now install PySpark, here we will install pyspark[sql] because it will allow us to deal with .gtf, .bed, .bam, and .sam files later.
!pip install pyspark[sql]
Sequence data from published research is available for download from SRA. Before you can use the sra file, you need to extract .fastq from it using the fastq-dump tool of SRA toolkit. If available, instead of SRA files, you may directly download .fastq, .fa.gz, or .fastq.gz formats, these files can be directly read with PySpark.
Be mindful of the file size.
At the core of PySpark is the resilient distributed dataset (RDD), it represents an immutable partitioned collection of your data that can be operated in parallel.
You begin by initializing a spark context.
import pyspark as sparkfrom pyspark import SparkConfsc = spark.SparkContext.getOrCreate(conf=set_conf())
It’s also a good practice to pass in a configuration that is optimal for your workflow.
Now it’s time to read and transform our data to RDD.
data = sc.textFile(path/to/fastq)# in case you have a list of sequencesdata = sc.parallelize(your_list)# lets take a look at the first read# each read in fastq is represented by 4lines data.take(4)
Let’s extract just the sequences from the data.
sequences = data.filter(lambda x: x.isalpha())sequences.count() # outputs the size of RDD - the number of reads# => 1843156
Take a look at the first four reads.
sequences.take(4)
How about finding the length of the reads? Takes only a line of code!
read_lengths = sequences.map(lambda seq: len(seq))read_lengths.take(10)
Next, compute the average length of the reads.
len_sum = read_lengths.reduce(lambda a, b: a+b)len_sum//read_lengths.count()# => 564
Finally, let’s do the base count. First, we will split the sequences to individual bases using list(), then we use flatMap to merge the lists to a single one. Then we go through individual bases and create a tuple with the first element as the base and ‘1’ as its value. Then we use reduceByKey to aggregate values by the first element — key — of the tuple.
base_count = sequences.flatMap(lambda seq: list(seq))\                       .map(lambda c: (c, 1)) \                       .reduceByKey(lambda a, b: a+b)base_count
We only care about A, T, G, and C here; the rest are artifacts. As you can see, these sequences are GC rich.
I have only scratched the surface here, and there are a lot of cool things you can do with PySpark. I will come back with more in another piece.
You can find the code examples in this colab notebook
I hope I have covered enough basics here to get you started. Talk to you next time.