pysam-developers/pysam — pull request #44 by chapmanb (2014-09-30 13:04:42+00:00)
Title: Fix issues with identifying max fields for a tuple index when input file contains empty elements

Andreas;
Retrieval of items from tabix indexed tab-delimited files will fail on files
with more than 3 empty columns. Here is a small code sample that demonstrates
the issue:

```python
import subprocess
import string

import pysam

def main():
# All work
test_retrieve(["chr1", "1", "20"] + ["a", "b"])
test_retrieve(["chr1", "1", "20"] + list(string.lowercase[:15]))
test_retrieve(["chr1", "1", "20"] + [""] * 3 + ["a", "b"])
# Will fail with 0.8.0 development version
test_retrieve(["chr1", "1", "20"] + [""] * 4 + ["a", "b"])
test_retrieve(["chr1", "1", "20"] + [""] * 20 + ["a", "b"])

def test_retrieve(fields):
test_file = "test.bed"
with open(test_file, "w") as out_handle:
out_handle.write("\t".join(fields) + "\n")
subprocess.check_call(["bgzip", "-f", test_file])
subprocess.check_call(["tabix", "-f", "-p", "bed", test_file + ".gz"])

tabix_test = pysam.Tabixfile(test_file + ".gz")
for out in tabix_test.fetch("chr1", 1, 10, parser=pysam.asTuple()):
print list(out)

if __name__ == "__main__":
main()
```
In 0.8.0 and the current development version, this will raise:

```
Traceback (most recent call last):
File "pysam_issue.py", line 24, in
main()
File "pysam_issue.py", line 10, in main
test_retrieve(["chr1", "1", "20"] + [""] * 4 + ["a", "b"])
File "pysam_issue.py", line 20, in test_retrieve
for out in tabix_test.fetch("chr1", 1, 10, parser=pysam.asTuple()):
File "ctabix.pyx", line 507, in pysam.ctabix.TabixIteratorParsed.__next__ (pysam/ctabix.c:5561)
File "ctabix.pyx", line 105, in pysam.ctabix.asTuple.parse (pysam/ctabix.c:2701)
File "TabProxies.pyx", line 149, in pysam.TabProxies.TupleProxy.copy (pysam/TabProxies.c:2692)
File "TabProxies.pyx", line 224, in pysam.TabProxies.TupleProxy.update (pysam/TabProxies.c:3180)
ValueError: row too large - more than 8 fields
```
This fix splits the string and uses the fields instead of trying to calculate
from number of bytes. That seems to be increasingly incorrect as you get
additional empty fields in the input tab delimited file.

Thanks much for looking at this and all your work on pysam.