samtools/samtools — pull request #26 by specht (2013-02-05 09:19:57+00:00)
Title: Added block size parameter for reduced I/O operations

## Summary

When running samtools on a cluster, the number of I/O syscalls becomes and important factor. An optional, larger buffer greatly improves overall speed because the number of filesystem calls can be reduced by a large amount when a slightly larger I/O buffer is used on each node.

For this purpose, I introduced a new option -z which can be used to specify the buffer size in kB for each input and output file. This currently only works with samtools view, sort and index. All remaining operations are not affected.

I measured the number of read/write syscalls using strace and a 126 MB SAM file:

$ strace -c ./samtools view [-z 4096] -Sbt hg19.fa A.sam > A-unsorted.bam

| read write
--------+------------
default | 8040 3274
-z 4096 | 25 11

$ strace -c ../samtools sort [-z 4096] A-unsorted.bam A-sorted

| read write
--------+------------
default | 3285 3274
-z 4096 | 17 7

$ strace -c ../samtools index [-z 4096] A-sorted.bam

| read write
--------+------------
default | 3290 231
-z 4096 | 16 1

I also verified that the resulting checksums are the same with and without the -z option, as expected.

## Implementation details

In order to make the buffer size varibale (g_block_size) globally available, I added globals.c and global.h which currently serve the sole purpose of hosting the variable and making it accessible. I am aware that this is probably not the best solution, but considering the fact that the variable is required in many places throughout the code and this is C, I hope the solution is ok.

Whenever a gzip file is opened via gzopen or gzdopen, gzbuffer is used to set the buffer size. Whenever a file is opened via fopen, a buffer is allocated via malloc, assigned to the FILE* via setvbuf, and later freed again.

Finally, I had to change the underlying implementation in knetfile.c from using file descriptors to using FILE* because as far as I know, buffering is not possible using file descriptors. For this purpose, I added FILE *fp alongside the int fd to be used when a local file is being read.

Please don't hestitate to contact me with questions or comments at micha.specht@gmail.com.

Cheers,
Micha.