agordon/fastx_toolkit — pull request #3 by KonstantinShemyak (2014-01-17 23:18:50+00:00)
Title: Fixed buffer overflow bug, caused by erroneous array boundaries

fastq_quality_filter segfaults on the following FASTQ file:

@
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
+
DDDfmnnei__pplWWWstXXX|hhaa~FFSDDDCDnaaRRYR`fkkdd{ss~wzz~~vooha^ax~~}}

It may result in another memory error than segmentation fault on your machine, but the fact is that src/fastq_quality_filter/fastq_quality_filter.c:get_percentile_quality attempts to write to an address out of the allocated space in line 120.

The reason is that the array quality_values[] has size MAX_QUALITY_VALUE - MIN_QUALITY_VALUE, but the index fastx->quality[i] - MIN_QUALITY_VALUE can be bigger than this. When fastx->quality[i] is being set, fastx.c checks that the value is <93 (hardcoded). 93 is bigger than MAX_QUALITY_VALUE, so the index is out of bounds.

The patch redefines [MIN|MAX]_QUALITY_VALUE from seemingly random -50/+50 to the values which are now checked for in fastx.c (-15 and 93), and replaces these hardcoded numbers with these constants.