How to download NGS data from the NCBI database | Towards Data Science

The Sequence Read Archive, or SRA, is NCBI’s database of storing Next Generation Sequence DNA data from sequencing experiments on any species, including humans. For those new to bioinformatics, it can be unclear how to access the data from this massive resource. It’s most likely you will be trying to access data from a research article you’ve just read, and the authors will have provide a Data Availability Statement that includes an accession number. This guide will assume you have the accession number ready. Bioinformatics software runs primarily on Unix based operating systems, and this is the case for the toolkit needed to download SRA data. You will need a computer running Linux or MacOS. This guide is specifically for MacOS, but the commands for MacOS and Linux are mostly the same, though they might require some minor tweaks between operating systems. Finally, NGS data is typically massive so you will need a decent amount of free storage on your machine. Depending on the species the dataset came from, full analysis of the NGS data can use upwards of 100Gb of memory. Just downloading a singe file should required no more than 50Gb though. Let’s start with downloading the software required for the job.
The SRA Toolkit contains all the programs we need to download and manipulate the data from SRA. To download it, navigate over to NCBI’s sra-tools GitHub page and download the appropriate file for your operating system. In your Downloads folder, double click the .tar file and navigate into the newly created folder. The programs we will be using for downloading the SRA data are located in the bin folder. We need to get the pathname of that bin folder and navigate into it on the command line. To do this, right click the bin folder, hold down the option key, and select ‘Copy “bin” as pathname’.
Now, open terminal and type in the following command, replacing <path/to/bin> by pasting in the pathname copied earlier:
cd <path/to/bin>
To check everything is in working order, run the following command:
./fastq-dump — stdout SRR390728 | head -n 8
You should get the following output:
@SRR390728.1 1 length=72CATTCTTCACGTAGTTCTCGAGCCTTGGTTTTCAGCGATGGAGAATGACTTTGACAAGCTGAGAGAAGNTNC+SRR390728.1 1 length=72;;;;;;;;;;;;;;;;;;;;;;;;;;;9;;665142;;;;;;;;;;;;;;;;;;;;;;;;;;;;;96&&&&(@SRR390728.2 2 length=72AAGTAGGTCTCGTCTGTGTTTTCTACGAGCTTGTGTTCCAGCTGACCCACTCCCTGGGTGGGGGGACTGGGT+SRR390728.2 2 length=72;;;;;;;;;;;;;;;;;4;;;;3;393.1+4&&5&&;;;;;;;;;;;;;;;;;;;;;<9;<;;;;;464262
We are now ready to download some data.
The first thing we need to do is find the Run number of the data, which is different to the accession number listed in the research article. To get the Run number, navigate to the NCBI SRA webpage, paste in your accession number from the research article and search. From the list, click the sample you are interested in, and under Runs, copy the number beginning with SRR followed by 6 or 7 digits. There is a lot of different accession type numbers on this page, but you have to select the right one to use in the next step.
Back to the command line, we are going to use the prefetch command to download the data plus any necessary files associated with it. To download the data run the following command, where <SRR0000000> is your Run number from above.
./prefetch <SRR0000000>
The file will download to the bin folder, so take a look to make sure it’s there after the download is complete and before moving on to the next step. Downloading could take a while depending on your internet speed and the size of the file being downloaded.
The prefetch program will download a .sra file, which is not a very useful file format. We need to convert this file into the standard NGS file format of FASTQ to do any work with it. We will use the fastq-dump tool to convert the data. There are quite a few options you can use with fastq-dump but if you are not sure which one to use, you can safely ignore them all for now. The --split-files option is likely the most useful one, it splits paired-end reads into separate files. To convert the .sra file into .fastq, make sure the file is in the bin folder and run the following command, where [options] are any relevant options and <SRR0000000> is the SRR number of your .sra file:
./fastq-dump [options] <SRR0000000>
The final FASTQ file will be dumped into the bin folder. This process could take a while depending on file size. After the conversion, we can visualise some of the bases in the file by using the head command piped into the less command.
head -10000 <filename> | less
You can go on to do any further processing with your FASTQ file now, including mapping to reference and variant analysis.