mdozmorov/RepeatSoaker — pull request #1 by glennsb (2014-03-21 14:35:00+00:00)
Title: Possible slightly faster overlap match position check

Alright, I've run the code through cProfile a bit (again I don't really know the Python mindest or environment,
so I'm just somewhat fumbling around in the dark).

### Original Code
```
$ head profile.txt
22334802 function calls (22327648 primitive calls) in 1122.736 seconds

Ordered by: tottime

ncalls tottime percall cumtime percall filename:lineno(function)
76972 843.205 0.011 843.205 0.011 RepeatSoaker.py:70()
1 230.053 230.053 1121.912 1121.912 RepeatSoaker.py:12(main)
5298131 33.416 0.000 37.682 0.000 BED.py:12(__iter__)
1 9.446 9.446 9.446 9.446 {method 'build' of 'BioTK.genome.index.RAMIndex' objects}
5300541 3.220 0.000 3.220 0.000 {method 'split' of 'str' objects}
```

The offending line 70 in that code is

```python
positions = set([p for p in positions$
if not ((p >= m_s) and (p <= m_e))])
```

So this is the masking bit, where it is doing a lot of what I assume is Pythons set comprehension stuff,
looping & creating a lot of sets. My naive attempt is to change from creating many sets to just a dictionary
and instead just setting the data to false if it is in the overlap. I think this is what the list stuff might
be after, but huge caveat again, I don't really do python so I might have that totally backwards

### Using Dicts

```
$ head profile4.txt
72852014 function calls (72844860 primitive calls) in 885.191 seconds

Ordered by: tottime

ncalls tottime percall cumtime percall filename:lineno(function)
1 373.463 373.463 884.901 884.901 RepeatSoaker.py:12(main)
1353054 319.760 0.000 319.760 0.000 RepeatSoaker.py:76()
1353058 70.529 0.000 70.529 0.000 {built-in method fromkeys}
2895137 37.339 0.000 37.547 0.000 csamtools.pyx:1295(write)
5298131 33.781 0.000 37.916 0.000 BED.py:12(__iter__)
2895137 19.649 0.000 19.649 0.000 {method 'search' of 'BioTK.genome.index.RAMIndex' objects}
1 9.490 9.490 9.490 9.490 {method 'build' of 'BioTK.genome.index.RAMIndex' objects}
2895139 5.592 0.000 6.708 0.000 csamtools.pyx:1561(__next__)
5601245 3.358 0.000 3.797 0.000 csamtools.pyx:2930(__get__)
5300541 3.087 0.000 3.087 0.000 {method 'split' of 'str' objects}
2895137 1.116 0.000 1.116 0.000 csamtools.pyx:159(makeAlignedRead)
2895137 1.044 0.000 39.608 0.000 {method 'write' of 'pysam.csamtools.Samfile' objects}
5299633 0.946 0.000 0.946 0.000 {method 'strip' of 'str' objects}
```

So, now the biggest hit is my bad bit to change the set back into another hash without the false values.
There is probably a much better way to count out the true positions left for the overlap

Anyway, in my small test case using just a chr1 it did shave off ~237.545 seconds of runtime according to those
top number of seconds in the profile head.

But again it is likely in no way the "python way" of things