arq5x/grabix — pull request #11 by chapmanb (2013-09-27 17:04:49+00:00)
Title: grabix index speed up: avoid double checking for EOF while identifying chunk boundaries

Aaron;
I'm indexing big FASTQ files (850 million lines for some initial tests) and found it taking an unexpectedly long time. My initial run went 9 hours before I gave up. Looking into the code it appears as if there is an extra check for EOF on each line which slows things way down. Removing this gives me clean indexing in 7 minutes on the test file. I'm modifying my CV now to add 'C skillz: demonstrated 100x performance improvements on production code'.