Repository: biopython/biopython

Issue:
SeqIO.index fails on empty ACCESSION lines in GenBank
When running `SeqIO.index` on a GenBank file that has an `ACCESSION` line without any content, `SeqIO.index` crashes with an `IndexError` like this:

```
Traceback (most recent call last):
  File "./check.py", line 6, in <module>
    rec = SeqIO.index(sys.argv[1], 'genbank')
  File "/home/kai/dev/libs/biopython/Bio/SeqIO/__init__.py", line 885, in index
    key_function, repr, "SeqRecord")
  File "/home/kai/dev/libs/biopython/Bio/File.py", line 294, in __init__
    for key, offset, length in offset_iter:
  File "/home/kai/dev/libs/biopython/Bio/SeqIO/_index.py", line 268, in __iter__
    key = line.rstrip().split()[1]
IndexError: list index out of range
```

The regular GenBank parser is handling these files just fine, it functionally reacts like there was no `ACCESSION` line.


Discussion:
Unfortunately, it looks like `SeqIO.index` is very insistent on not using the ID available in the `LOCUS` line, so the behaviour here differs. I'm a bit at a loss on how to fix this properly.
That said, we should at least trigger the same `ValueError` the code throws if there is no `ACCESSION` or `VERSION` line.
It probably does not matter given the ``.rstrip()`` but can you share an example - is it just ``ACCESSION\n`` in these cases, or might there be trailing whitespace?
Do your files have a ``VERSION`` line which would be used in preference to the (blank) ``ACCESSION`` line?

If neither is missing then from memory the parser would use the ``LOCUS`` line - but the indexer does not have that implemented.
In this particular case, the file has trailing whitespace, and no `VERSION` line. Let me reproduce with a similar file I can share.

This one has the same issue:
[empty_accession.gbk.txt](https://github.com/biopython/biopython/files/1180176/empty_accession.gbk.txt)

Short term "fix" would be:

```
$ git diff
diff --git a/Bio/SeqIO/_index.py b/Bio/SeqIO/_index.py
index 952b4856f..6a1b2fdd5 100644
--- a/Bio/SeqIO/_index.py
+++ b/Bio/SeqIO/_index.py
@@ -265,7 +265,11 @@ class GenBankRandomAccess(SequentialSeqFileRandomAccess):
                     start_offset = end_offset
                     break
                 elif line.startswith(accession_marker):
-                    key = line.rstrip().split()[1]
+                    try:
+                        key = line.rstrip().split()[1]
+                    except ValueError:
+                        # Missing any accession
+                        pass
                 elif line.startswith(version_marker):
                     version_id = line.rstrip().split()[1]
                     if version_id.count(b".") == 1 and version_id.split(b".")[1].isdigit():
```

That should at least trigger ValueError ``Did not find ACCESSION/VERSION lines`` although there would actually be an incomplete ACCESSION line.
The real fix is to implement the exact same ``LOCUS`` line fall back implemented by the GenBank parser.

(Given a clean slate for efficient indexing I would have used the ``LOCUS`` line entry as the record's ``.id`` but that was not what the parser did when I came to write the indexing)
You need to catch an `IndexError`, not a `ValueError`, though.
I'd have submitted a patch in that direction already, but I can't figure out where to put the unit test for that. I am confused by `test_SeqIO_index.py`.
That said, so far I relied on having another part of my code sanity checking the GenBank file via `parse()` prior to indexing it where this blew up, and I guess I need more defense in depth anyway.
Good point on the exception, how about:

```
$ git diff
diff --git a/Bio/SeqIO/_index.py b/Bio/SeqIO/_index.py
index 952b4856f..7de4e5063 100644
--- a/Bio/SeqIO/_index.py
+++ b/Bio/SeqIO/_index.py
@@ -260,12 +260,16 @@ class GenBankRandomAccess(SequentialSeqFileRandomAccess):
                 if marker_re.match(line) or not line:
                     if not key:
                         raise ValueError(
-                            "Did not find ACCESSION/VERSION lines")
+                            "Did not find usable ACCESSION/VERSION lines")
                     yield _bytes_to_string(key), start_offset, length
                     start_offset = end_offset
                     break
                 elif line.startswith(accession_marker):
-                    key = line.rstrip().split()[1]
+                    try:
+                        key = line.rstrip().split()[1]
+                    except ValueError:
+                        # Missing any accession
+                        pass
                 elif line.startswith(version_marker):
                     version_id = line.rstrip().split()[1]
                     if version_id.count(b".") == 1 and version_id.split(b".")[1].isdigit():
```

Regarding ``test_SeqIO_index.py`` the first half is fairly traditional unittests (some wrapped for use when SQLite3 is missing), then there is a list for files and formats from which tests are generated dynamically.

Simple option add your new pathological GenBank file to this list:
https://github.com/biopython/biopython/blob/biopython-170/Tests/test_SeqIO_index.py#L611


How do I tell that test that I expect a ValueError then?
Umm. Good point, I hadn't thought that though. Those main tests are for working files.

Right now it would have be a special case test method, something like:

```python
class Problems(unitest.TestCase):
    def test_genbank_no_acc_ver(self):
        filename = "..."
        self.assertRaises(ValueError, SeqIO.index, filename, "gb")
```

It would be better to fix the indexer for this case, which I guess is something for me to tackle if I can find a quiet moment.
I agree that the proper fix would be in the indexer, but at least now we return the same exception type for all invalid inputs.
Thanks for #1346 - clear exceptions here is a worthwhile step forward.

Resolving patch:
diff --git a/Bio/SeqIO/_index.py b/Bio/SeqIO/_index.py
--- a/Bio/SeqIO/_index.py
+++ b/Bio/SeqIO/_index.py
@@ -252,15 +252,20 @@ def __iter__(self):
         while marker_re.match(line):
             # We cannot assume the record.id is the first word after LOCUS,
             # normally the first entry on the VERSION or ACCESSION line is used.
-            key = None
+            # However if both missing, GenBank parser falls back on LOCUS entry.
+            try:
+                key = line[5:].split(None, 1)[0]
+            except ValueError:
+                # Warning?
+                # No content in LOCUS line
+                key = None
             length = len(line)
             while True:
                 end_offset = handle.tell()
                 line = handle.readline()
                 if marker_re.match(line) or not line:
                     if not key:
-                        raise ValueError(
-                            "Did not find usable ACCESSION/VERSION lines")
+                        raise ValueError("Did not find usable ACCESSION/VERSION/LOCUS lines")
                     yield _bytes_to_string(key), start_offset, length
                     start_offset = end_offset
                     break


Test patch:
diff --git a/Tests/test_SeqIO_index.py b/Tests/test_SeqIO_index.py
--- a/Tests/test_SeqIO_index.py
+++ b/Tests/test_SeqIO_index.py
@@ -295,33 +295,6 @@ def test_some_abs(self):
                        expt_sff_files)
 
 
-class Problems(unittest.TestCase):
-    """Corner cases where we expect indexing to fail.
-
-    Goal of these tests is to make sure they fail the right way.
-    """
-
-    def setUp(self):
-        os.chdir(CUR_DIR)
-        h, self.index_tmp = tempfile.mkstemp("_idx.tmp")
-        os.close(h)
-
-    def tearDown(self):
-        os.chdir(CUR_DIR)
-        if os.path.isfile(self.index_tmp):
-            os.remove(self.index_tmp)
-
-    def test_genbank_empty_accession(self):
-        """Test an empty ACCESSION line causes a ValueError."""
-        filename = "GenBank/empty_accession.gbk"
-        self.assertRaises(ValueError, SeqIO.index, filename, 'genbank')
-
-    def test_genbank_empty_version(self):
-        """Test an empty VERSION line causes a ValueError."""
-        filename = "GenBank/empty_version.gbk"
-        self.assertRaises(ValueError, SeqIO.index, filename, 'genbank')
-
-
 class IndexDictTests(unittest.TestCase):
     """Cunning unit test where methods are added at run time."""
 
@@ -664,6 +637,8 @@ def test_duplicates_to_dict(self):
     ("GenBank/NC_005816.fna", "fasta", generic_dna),
     ("GenBank/NC_005816.gb", "gb", None),
     ("GenBank/cor6_6.gb", "genbank", None),
+    ("GenBank/empty_accession.gbk", "gb", None),
+    ("GenBank/empty_version.gbk", "gb", None),
     ("IntelliGenetics/vpu_nucaligned.txt", "ig", generic_nucleotide),
     ("IntelliGenetics/TAT_mase_nuc.txt", "ig", None),
     ("IntelliGenetics/VIF_mase-pro.txt", "ig", generic_protein),