Question: Hi,

I am getting following error while doing denovo assembly using SPAdes on a linux with 15 GB RAM and more than 50GB space left in hard disk. The size of two fastq files being used as input is about 2.4 GB and 2 GB.

ERROR K-mer Counting
The reads contain too many k-mers to fit into available memory limit. Increase memory limit and restart

Spades.log:

Command line: spades.py --careful -o WT_ -1 firstfile.fq -2 secondfile.fq -m 10

System information:
SPAdes version: 3.5.0
Python version: 2.7.12
OS: Linux-4.4.0-59-generic-x86_64-with-Ubuntu-16.04-xenial

Output dir: SPAdes-3.5.0-Linux/bin/WT_
Mode: read error correction and assembling
Debug mode is turned OFF

Dataset parameters:
Multi-cell mode (you should set '--sc' flag if input data was obtained with MDA (single-cell) technology
Reads:
Library number: 1, library type: paired-end
orientation: fr
left reads: ['firstfile.fq']
right reads: ['secondfile.fq']
interlaced reads: not specified
single reads: not specified
Read error correction parameters:
Iterations: 1
PHRED offset will be auto-detected
Corrected reads will be compressed (with gzip)
Assembly parameters:
k: automatic selection based on read length
Mismatch careful mode is turned ON
Repeat resolution is enabled
MismatchCorrector will be used
Coverage cutoff is turned OFF
Other parameters:
Dir for temp files: tmp
Threads: 16
Memory limit (in Gb): 10

======= SPAdes pipeline started. Log can be found here: SPAdes-3.5.0-Linux/bin/WT_/spades.log

===== Read error correction started.

== Running read error correction tool: SPAdes-3.5.0-Linux/bin/hammer SPAdes-3.5.0-Linux/bin/WT_/corrected/configs/config.info

0:00:00.000 4M / 4M INFO General (main.cpp : 82) Loading config from SPAdes-3.5.0-Linux/bin/WT_/corrected/configs/config.info
0:00:00.000 4M / 4M INFO General (memory_limit.hpp : 42) Memory limit set to 10 Gb
0:00:00.001 4M / 4M INFO General (main.cpp : 91) Trying to determine PHRED offset
0:00:00.001 4M / 4M INFO General (main.cpp : 97) Determined value is 33
0:00:00.002 4M / 4M INFO General (hammer_tools.cpp : 36) Hamming graph threshold tau=1, k=21, subkmer positions = [ 0 10 ]
=== ITERATION 0 begins ===
0:00:00.002 4M / 4M INFO K-mer Index Building (kmer_index.hpp : 467) Building kmer index
0:00:00.002 4M / 4M INFO K-mer Splitting (kmer_data.cpp : 127) Splitting kmer instances into 128 buckets. This might take a while.
0:00:00.002 4M / 4M INFO General (file_limit.hpp : 29) Open file limit set to 1024
0:00:00.002 4M / 4M INFO K-mer Splitting (kmer_data.cpp : 145) Memory available for splitting buffers: 0.416504 Gb
0:00:00.002 4M / 4M INFO K-mer Splitting (kmer_data.cpp : 153) Using cell size of 436736
0:00:00.857 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 167) Processing firstfile.fq
0:00:18.381 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 813597 reads
0:00:38.048 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 1673452 reads
0:00:57.634 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 2519299 reads
0:01:16.964 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 3305418 reads
0:01:37.462 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 4168421 reads
0:01:50.922 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 4493764 reads
0:01:50.922 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 167) Processing secondfile.fq
0:02:08.666 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 5591263 reads
0:02:35.935 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 6636651 reads
0:03:20.730 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 176) Processed 8752362 reads
0:03:25.466 3G / 3G INFO K-mer Splitting (kmer_data.cpp : 181) Processed 8987528 reads
0:03:25.620 32M / 3G INFO General (kmer_index.hpp : 345) Starting k-mer counting.
0:03:53.603 32M / 3G INFO General (kmer_index.hpp : 351) K-mer counting done. There are 418968448 kmers in total.
0:03:53.603 32M / 3G INFO General (kmer_index.hpp : 353) Merging temporary buckets.
0:04:11.857 32M / 3G INFO K-mer Index Building (kmer_index.hpp : 476) Building perfect hash indices
0:06:34.813 160M / 7G INFO General (kmer_index.hpp : 371) Merging final buckets.
0:06:50.161 160M / 7G INFO K-mer Index Building (kmer_index.hpp : 515) Index built. Total 144936940 bytes occupied (2.7675 bits per kmer).
0:06:50.161 160M / 7G ERROR K-mer Counting (kmer_data.cpp : 261) The reads contain too many k-mers to fit into available memory limit. Increase memory limit and restart

== Error == system call for: "['SPAdes-3.5.0-Linux/bin/hammer', 'SPAdes-3.5.0-Linux/bin/WT_/corrected/configs/config.info']" finished abnormally, err code: 255

In case you have troubles running SPAdes, you can write to spades.support@bioinf.spbau.ru
Please provide us with params.txt and spades.log files from the output directory.

params.txt

Command line: spades.py --careful -o WT_ -1 firstfile.fq -2 secondfile.fq -m 10

System information:
SPAdes version: 3.5.0
Python version: 2.7.12
OS: Linux-4.4.0-59-generic-x86_64-with-Ubuntu-16.04-xenial

Output dir: SPAdes-3.5.0-Linux/bin/
Mode: read error correction and assembling
Debug mode is turned OFF

Dataset parameters:
Multi-cell mode (you should set '--sc' flag if input data was obtained with MDA (single-cell) technology
Reads:
Library number: 1, library type: paired-end
orientation: fr
left reads: ['firstfile.fq']
right reads: ['secondfile.fq']
interlaced reads: not specified
single reads: not specified
Read error correction parameters:
Iterations: 1
PHRED offset will be auto-detected
Corrected reads will be compressed (with gzip)
Assembly parameters:
k: automatic selection based on read length
Mismatch careful mode is turned ON
Repeat resolution is enabled
MismatchCorrector will be used
Coverage cutoff is turned OFF
Other parameters:
Dir for temp files: SPAdes-3.5.0-Linux/bin/WT_/tmp
Threads: 16
Memory limit (in Gb): 10

Thx for the help.

Cheers,
Ambi.

Answer: 15GB RAM is tiny for most purposes of genome assembly. Also, you're using an old version of Spades; newer versions are more memory-efficient, so you should upgrade.

If for some reason you only have 15GB RAM available, you can greatly reduce the amount of time and memory Spades will need using the BBMap package to discard low-abundance kmers. Generally, I'd recommend a pipeline something like this (specifics would require much more information about your data):

#Trim adapters
bbduk.sh in=r1.fq in2=r2.fq out=trimmed.fq ktrim=r k=23 mink=11 hdist=1 ref=/bbmap/resources/adapters.fa tbo tpe

#Normalize
bbnorm.sh in=trimmed.fq out=normalized.fq target=100 min=5

#Assemble
spades.py -k 21,41,71,101,127 -o out -12 normalized.fq

As shenwei indicated, you can also reduce memory consumption by using shorter kmers. But if you are actually interested in getting the best possible assembly, you shouldn't do that; rather, you should find a computer with more memory.