Trying pydablooms -> pybloomfilter (seems slower)

Benchline on a Mac Pro using pydablooms, backed by the set filter to remove
the false positives:

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.01
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 2.7s (of which 1.9s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.3s (of which 3.8s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 8.0s (of which 5.6s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 10.6s (of which 7.5s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 13.3s (of which 9.4s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 15.9s (of which 11.3s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 18.6s (of which 13.2s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 21.3s (of which 15.0s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 23.9s (of which 16.9s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 26.6s (of which 18.8s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 29.3s (of which 20.7s in filter)
...
Processed 268400000 reads, kept 2138512 (0.8%), taken 7146.7s (of which 5054.5s in filter)
Processed 268500000 reads, kept 2139344 (0.8%), taken 7149.4s (of which 5056.4s in filter)
Running filter took 5057.2s, overhead 2093.3s, total 7150.5s
Kept 2139634 out of 268540706 reads

real119m10.753s
user131m30.556s
sys1m14.834s

And now using pybloomfilter with the same code and settings otherwise,

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6_pybloomfilter.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.01
Building filters took 0.6s
Processed 100000 reads, kept 838 (0.8%), taken 15.3s (of which 14.5s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 30.6s (of which 29.0s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 45.9s (of which 43.6s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 61.2s (of which 58.1s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 76.5s (of which 72.6s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 91.9s (of which 87.2s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 107.2s (of which 101.8s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 122.5s (of which 116.3s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 137.8s (of which 130.9s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 153.1s (of which 145.4s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 168.4s (of which 159.9s in filter)
...

Looks slower, quite a lot slower :(
Revert "Trying pydablooms -> pybloomfilter (seems slower)"

This reverts commit 8f3d9d18e277219a70e511f9fa0f10066aaf2773.
Trying pydablooms -> hydra (seems slower)

Benchline on a Mac Pro using pydablooms, backed by the set filter to remove
the false positives:

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mit
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.01
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 2.7s (of which 1.9s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.3s (of which 3.8s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 8.0s (of which 5.6s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 10.6s (of which 7.5s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 13.3s (of which 9.4s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 15.9s (of which 11.3s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 18.6s (of which 13.2s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 21.3s (of which 15.0s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 23.9s (of which 16.9s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 26.6s (of which 18.8s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 29.3s (of which 20.7s in filter)
...
Processed 268400000 reads, kept 2138512 (0.8%), taken 7146.7s (of which 5054.5s in filter)
Processed 268500000 reads, kept 2139344 (0.8%), taken 7149.4s (of which 5056.4s in filter)
Running filter took 5057.2s, overhead 2093.3s, total 7150.5s
Kept 2139634 out of 268540706 reads

real119m10.753s
user131m30.556s
sys1m14.834s

Using hydra 1.0, but otherwise the same script:

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6_hydra.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.01
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 5.4s (of which 4.6s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 10.7s (of which 9.2s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 16.1s (of which 13.8s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 21.4s (of which 18.4s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 26.8s (of which 23.0s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 32.2s (of which 27.6s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 37.5s (of which 32.2s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 42.9s (of which 36.8s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 48.3s (of which 41.4s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 53.6s (of which 46.0s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 59.0s (of which 50.6s in filter)
Processed 1200000 reads, kept 9580 (0.8%), taken 64.3s (of which 55.2s in filter)
Processed 1300000 reads, kept 10374 (0.8%), taken 69.7s (of which 59.8s in filter)
Processed 1400000 reads, kept 11142 (0.8%), taken 75.1s (of which 64.4s in filter)
...

About twice the time needed.
Revert "Trying pydablooms -> hydra (seems slower)"

This reverts commit ee306e8724700475b829fac1e1ca9268d0a81903.