Try and do this at the paired-read level
Debug logging of progress

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.0005
Building filters took 0.1s
Processed 100000 reads, kept 5072 (5.1%), taken 2.7s (of which 1.9s in filter)
Processed 200000 reads, kept 10122 (5.1%), taken 5.5s (of which 3.9s in filter)
Processed 300000 reads, kept 15198 (5.1%), taken 8.2s (of which 5.8s in filter)
Processed 400000 reads, kept 20166 (5.0%), taken 11.0s (of which 7.8s in filter)
Processed 500000 reads, kept 25076 (5.0%), taken 13.7s (of which 9.7s in filter)
Processed 600000 reads, kept 30084 (5.0%), taken 16.4s (of which 11.7s in filter)
Processed 700000 reads, kept 35106 (5.0%), taken 19.1s (of which 13.6s in filter)
Processed 800000 reads, kept 39946 (5.0%), taken 21.8s (of which 15.5s in filter)
Processed 900000 reads, kept 45000 (5.0%), taken 24.6s (of which 17.5s in filter)
Processed 1000000 reads, kept 49980 (5.0%), taken 27.3s (of which 19.4s in filter)
Processed 1100000 reads, kept 54774 (5.0%), taken 30.0s (of which 21.4s in filter)
...
Turn on the full set filter again (for timing)

This is using both the bloom and the set filter:

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.0005
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 2.7s (of which 2.0s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.4s (of which 3.9s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 8.2s (of which 5.9s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 10.9s (of which 7.9s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 13.6s (of which 9.9s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 16.3s (of which 11.8s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 19.0s (of which 13.8s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 21.8s (of which 15.8s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 24.5s (of which 17.7s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 27.2s (of which 19.7s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 29.9s (of which 21.7s in filter)
...

For comparison, the times using just the set filter is surprisingly close:

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.0005
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 2.9s (of which 2.1s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.8s (of which 4.3s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 8.7s (of which 6.4s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 11.6s (of which 8.5s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 14.5s (of which 10.7s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 17.4s (of which 12.8s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 20.3s (of which 14.9s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 23.2s (of which 17.1s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 26.2s (of which 19.2s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 29.1s (of which 21.3s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 32.0s (of which 23.5s in filter)
...

Previous times using just the bloom filter are very similar, but keep 5% of the
reads rather than only 1%:

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Using /var/folders/t_/q00rqqr15lg950n1c8r2yxf00000gn/T/bloom-Z2g2wa.bin
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.0005
Building filters took 0.1s
Processed 100000 reads, kept 5072 (5.1%), taken 2.7s (of which 1.9s in filter)
Processed 200000 reads, kept 10122 (5.1%), taken 5.5s (of which 3.9s in filter)
Processed 300000 reads, kept 15198 (5.1%), taken 8.2s (of which 5.8s in filter)
Processed 400000 reads, kept 20166 (5.0%), taken 11.0s (of which 7.8s in filter)
Processed 500000 reads, kept 25076 (5.0%), taken 13.7s (of which 9.7s in filter)
Processed 600000 reads, kept 30084 (5.0%), taken 16.4s (of which 11.7s in filter)
Processed 700000 reads, kept 35106 (5.0%), taken 19.1s (of which 13.6s in filter)
Processed 800000 reads, kept 39946 (5.0%), taken 21.8s (of which 15.5s in filter)
Processed 900000 reads, kept 45000 (5.0%), taken 24.6s (of which 17.5s in filter)
Processed 1000000 reads, kept 49980 (5.0%), taken 27.3s (of which 19.4s in filter)
Processed 1100000 reads, kept 54774 (5.0%), taken 30.0s (of which 21.4s in filter)
...
Explore Bloom error rate setting and speed (in double filter).

A very high error rate (50%) is slower (and far more false positives must
be checked via the set filter), somewhere between 1% and 5% seems best on
this test set.

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.5
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 3.0s (of which 2.2s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.9s (of which 4.4s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 8.9s (of which 6.5s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 11.8s (of which 8.7s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 14.8s (of which 10.9s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 17.7s (of which 13.1s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 20.7s (of which 15.3s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 23.6s (of which 17.4s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 26.6s (of which 19.6s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 29.5s (of which 21.8s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 32.4s (of which 24.0s in filter)
...

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.01
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 2.7s (of which 1.9s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.3s (of which 3.8s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 8.0s (of which 5.6s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 10.6s (of which 7.5s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 13.3s (of which 9.4s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 15.9s (of which 11.3s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 18.5s (of which 13.2s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 21.2s (of which 15.0s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 23.8s (of which 16.9s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 26.5s (of which 18.8s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 29.2s (of which 20.7s in filter)
...

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.05
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 2.6s (of which 1.9s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.3s (of which 3.7s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 7.9s (of which 5.6s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 10.5s (of which 7.4s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 13.1s (of which 9.3s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 15.7s (of which 11.1s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 18.3s (of which 13.0s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 20.9s (of which 14.8s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 23.5s (of which 16.7s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 26.1s (of which 18.5s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 28.7s (of which 20.4s in filter)
...

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas > ~/data3tb/Globodera_pallida/Gp_mito_2012_Sep/mito_bloomed_7261_6.sam
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.005
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 2.7s (of which 1.9s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.4s (of which 3.8s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 8.0s (of which 5.7s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 10.8s (of which 7.7s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 13.4s (of which 9.6s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 16.1s (of which 11.5s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 18.8s (of which 13.4s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 21.5s (of which 15.3s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 24.2s (of which 17.2s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 26.9s (of which 19.1s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 29.5s (of which 21.0s in filter)
...

$ time samtools view ~/data3tb/Globodera_pallida/ena_reads/ERA127/ERA127534/bam/7261_6.bam | ./blooming_reads.py -f sam -l ~/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mit
Hashing linear references in /Users/pjcock/data3tb/Globodera_pallida/Gp_mito_2009_May/all_mito.fas
Set and bloom filter of 35-mers created (79552 k-mers considered, 53891 unique)
Using Bloom filter with capacity 53891 and error rate 0.0005
Building filters took 0.1s
Processed 100000 reads, kept 838 (0.8%), taken 2.7s (of which 2.0s in filter)
Processed 200000 reads, kept 1686 (0.8%), taken 5.4s (of which 3.9s in filter)
Processed 300000 reads, kept 2492 (0.8%), taken 8.2s (of which 5.9s in filter)
Processed 400000 reads, kept 3316 (0.8%), taken 10.9s (of which 7.9s in filter)
Processed 500000 reads, kept 4088 (0.8%), taken 13.6s (of which 9.9s in filter)
Processed 600000 reads, kept 4824 (0.8%), taken 16.3s (of which 11.8s in filter)
Processed 700000 reads, kept 5648 (0.8%), taken 19.0s (of which 13.8s in filter)
Processed 800000 reads, kept 6400 (0.8%), taken 21.8s (of which 15.8s in filter)
Processed 900000 reads, kept 7226 (0.8%), taken 24.5s (of which 17.7s in filter)
Processed 1000000 reads, kept 8018 (0.8%), taken 27.2s (of which 19.7s in filter)
Processed 1100000 reads, kept 8804 (0.8%), taken 29.9s (of which 21.7s in filter)
...