Package: fastq-filter
fastq-filterA fast FASTQ filter program.Fastq-filter correctly takes into account that quality scores are log scores
when calculating the mean. It also provides an option to filter on average
error rate directly.FASTQ Q=30 stands for an average error rate of 0.001, Q=20 for 0.01 and Q=10
for 0.1. This is not very intuitive. Q=20 has 10 times more errors than Q=30
though the numbers (20 and 30) do little to convey this difference. Using
0.01 and 0.001 correctly conveys that these error rates are an order of
magnitude apart. This also means that the phred scores cannot be naively
averaged. Q=10 and Q=30 do not average Q=20. The actual average error rate
is (0.001 + 0.1) / 2 = 0.0505. Roughly 1 in 20. Q=20 means 0.01: 1 in 100.
By naively averaging the quality is overestimated by a factor of 5! This
means any tool that averages naively is unusable in practice.Unfortunately many tools do this. fastq-filter was written to provide a very
fast filtering solution so the correct filtering can be applied at a very low
cost.InstallationWith pip:pip installfastq-filterFor the latest development version:pip installgit+https://github.com/LUMC/fastq-filterWith condaconda install-cconda-forge-cbiocondafastq-filterUsageSingle fastq files can be filtered with:fastq-filter -e 0.001 -o output.fastq input.fastqMultiple fastq files can be filtered with:fastq-filter -e 0.001 -o r1_filtered.fastq.gz -o r2_filtered.fastq.gz r1.fastq.gz r2.fastq.gzFastq-filter ensures the output is in sync. It is not limited to two inputs
so alsoR1.fq,R2.fqandR3.fqcan be filtered together.In the following section ‘pair’ is used to note when 2 or more FASTQ records are
evaluated. When multiple FASTQ files are given the filters behave as follows:average error rate: The average of the combined phred scores is used.median quality: The median of the combined phred scores is used.Minimum length: at least one of the records of the pair must meet the minimum length.Maximum length: None of the records in the pair must exceed the maximum length.The rationale for the length filters is that R1 and R2 both sequence the same
molecule and the canonical length is the longest of both.usage: fastq-filter [-h] [-o OUTPUT] [-l MIN_LENGTH] [-L MAX_LENGTH]
                    [-e AVERAGE_ERROR_RATE] [-q MEAN_QUALITY]
                    [-Q MEDIAN_QUALITY] [-c COMPRESSION_LEVEL] [--verbose]
                    [--quiet]
                    input [input ...]

Filter FASTQ files on various metrics.

positional arguments:
  input                 Input FASTQ files. Compression format automatically
                        detected. Use - for stdin.

optional arguments:
  -h, --help            show this help message and exit
  -o OUTPUT, --output OUTPUT
                        Output FASTQ files. Compression format automatically
                        determined by file extension. Flag can be used
                        multiple times. An output must be given for each
                        input. Default: stdout.
  -l MIN_LENGTH, --min-length MIN_LENGTH
                        The minimum length for a read.
  -L MAX_LENGTH, --max-length MAX_LENGTH
                        The maximum length for a read.
  -e AVERAGE_ERROR_RATE, --average-error-rate AVERAGE_ERROR_RATE
                        The minimum average per base error rate.
  -q MEAN_QUALITY, --mean-quality MEAN_QUALITY
                        Average quality. Same as the '--average-error-rate'
                        option but specified with a phred score. I.e '-q 30'
                        is equivalent to '-e 0.001'.
  -Q MEDIAN_QUALITY, --median-quality MEDIAN_QUALITY
                        The minimum median phred score.
  -c COMPRESSION_LEVEL, --compression-level COMPRESSION_LEVEL
                        Compression level for the output files. Relevant when
                        output files have a .gz extension. Default: 2
  --verbose             Report stats on individual filters.
  --quiet               Turn of logging output.Optimizationsfastq-filter has used the following optimizations to be fast:Multiple filters can applied simultaneously to minimize IO.fastq-filter can be used in pipes to minimize IOThe python filter function is used. Which is a a shorthand for python code
that would otherwise need to be interpreted.The mean and median quality algorithms are implemented in C with bindings to
Python.The mean quality algorithm uses a lookup table since there are only 93
possible phred scores encoded in FASTQ. That saves a lot of power
calculations to calculate the probabilities.The median quality algorithm implements a counting sort, which is really
fast but not applicable for generic data. Since FASTQ qualities are uniquely
suited for a counting sort, median calculation can be performed very quickly.dnaiois used as FASTQ parser.  This
parses the FASTQ files with a parser written in Cython.xopenis used to read and write
files. This allows for support of gzip compressed files which are opened
usingpython-isalwhich
reads gzip files 2 times faster and writes gzip files 5 times faster than
the pythongzipmodule implementation.