Repository: biopython/biopython

Issue:
Friendlier traceback for SeqIO.index when dealing with non-BGZIP compressed file
### Setup

I am reporting a problem with Biopython version, Python version, and operating
system as follows:

```
$ python --version
Python 3.6.5
$ python -c "import Bio; print(Bio.__version__)"
1.71.dev0
$ uname -a
Darwin localhost 16.7.0 Darwin Kernel Version 16.7.0: Tue Jan 30 11:27:06 PST 2018; root:xnu-3789.73.11~1/RELEASE_X86_64 x86_64
```

And the following backtrace when using `SeqIO.index` against a GZIP-compressed human reference genome:

```python
  File "./src/project/project_coords.py", line 136, in main
    record_dict = SeqIO.index(genome_build, "fasta")
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/Bio/SeqIO/__init__.py", line 891, in index
    key_function, repr, "SeqRecord")
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/Bio/File.py", line 308, in __init__
    for key, offset, length in offset_iter:
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/Bio/SeqIO/_index.py", line 211, in __iter__
    yield _bytes_to_string(id), start_offset, length
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/Bio/_py3k/__init__.py", line 59, in <lambda>
    _bytes_to_string = lambda b: b.decode()  # bytes to unicode string
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf7 in position 5: invalid start byte
```

### Expected behaviour

An informative error message that GZIP encoding has been detected instead of BGFZ-indexable binary file, as supported by the `.index()` method.

This explanation is in the docs, but it’s not immediately clear to the programmer as to why the method fails here. Being more explicit might help the programmer immediately realize the underlying issue and move on.

### Actual behaviour

`UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf7 in position 5: invalid start byte` is raised when feeding in a GZIP-compressed `hg38.fa.gz`.

From the docs in http://biopython.org/DIST/docs/api/Bio.SeqIO-module.html:

"If the file is BGZF compressed, this is detected automatically. Ordinary GZIP files are not supported"

```python
$ time ./src/project/project_coords.py data/external/hg38.fa.gz
Traceback (most recent call last):
  File "./src/project/project_coords.py", line 152, in <module>
    main()
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "./src/project/project_coords.py", line 136, in main
    record_dict = SeqIO.index(genome_build, "fasta")
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/Bio/SeqIO/__init__.py", line 891, in index
    key_function, repr, "SeqRecord")
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/Bio/File.py", line 308, in __init__
    for key, offset, length in offset_iter:
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/Bio/SeqIO/_index.py", line 211, in __iter__
    yield _bytes_to_string(id), start_offset, length
  File "/Users/romanvg/.miniconda/envs/projects/lib/python3.6/site-packages/Bio/_py3k/__init__.py", line 59, in <lambda>
    _bytes_to_string = lambda b: b.decode()  # bytes to unicode string
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf7 in position 5: invalid start byte
```

### Steps to reproduce

Just use `SeqIO.index()` against a `gzip hg38.fa` reference genome.


Discussion:
Sounds sensible, I'm surprised no one else reported this to date.

Probably the fix would be in private function ``_open_for_random_access()`` under ``Bio.File``,

https://github.com/biopython/biopython/blob/biopython-171/Bio/File.py#L107

This could sniff the for the GZIP magic two bytes, then if present attempt to open as BGZF. If that fails, we have a non-BGZF gzipped file, and can raise a specific exception. If the magic marker is missing, assume (as now) this is a non-compressed file and return the default handle.

Do you want to work on this and submit a pull request?
I added autodetection of ordinary gzipped files in _open_for_random_access method, but don't you think than current behavior of sequential access parsing methods (SeqIO.parse, SeqIO.read) and random-accessed methods (SeqIO.index, SeqIO.index_db) is not well consistent and may be confusing.
* random access methods use _open_for_random_access method that supports compessed method autodetection, but for sequential methods you must specify compression exactly via passing handler to compressed file gained via zgip.open. So you're unable to just pass path to grip-compressed file to Seq.parse. But you can do it (after merge the pull request) with Seq.index
* sequential access methos use as_handle method, that supports both file path and file handle. so you can pass file-like object like io.BytesIO to SeqIO.parse but not in SeqIO.index.

Don't you think that it will be better to eliminate _open_for_random_access method and instead use as_handle method for both sequential and random-access methods and improve as_handle method in a way that if file name is provided that it can autodetect compressed file. I guess it will make API a bit more consistent
No, the whole point of ``_open_for_random_access`` is to handle ordinary files or BGZF files for indexing (but NOT ordinary gzip files - here it should ideally give a clear error message). The indexing code assumes and expects a file on disk, there is no reason to support an in-memory file-like object like BytesIO.

For the separate issue of allowing parsing of gzipped files in SeqIO, AlignIO, SearchIO see #1686.

Resolving patch:
diff --git a/Bio/File.py b/Bio/File.py
--- a/Bio/File.py
+++ b/Bio/File.py
@@ -109,15 +109,26 @@ def _open_for_random_access(filename):
 
     This functionality is used by the Bio.SeqIO and Bio.SearchIO index
     and index_db functions.
+
+    If the file is gzipped but not BGZF, a specific ValueError is raised.
     """
     handle = open(filename, "rb")
-    from . import bgzf
-    try:
-        return bgzf.BgzfReader(mode="rb", fileobj=handle)
-    except ValueError as e:
-        assert "BGZF" in str(e)
-        # Not a BGZF file after all, rewind to start:
-        handle.seek(0)
+    magic = handle.read(2)
+    handle.seek(0)
+
+    if magic == b"\x1f\x8b":
+        # This is a gzipped file, but is it BGZF?
+        from . import bgzf
+        try:
+            # If it is BGZF, we support that
+            return bgzf.BgzfReader(mode="rb", fileobj=handle)
+        except ValueError as e:
+            assert "BGZF" in str(e)
+            # Not a BGZF file after all,
+            handle.close()
+            raise ValueError("Gzipped files are not suitable for indexing, "
+                             "please use BGZF (blocked gzip format) instead.")
+
     return handle
 
 


Test patch:
diff --git a/Tests/test_File.py b/Tests/test_File.py
--- a/Tests/test_File.py
+++ b/Tests/test_File.py
@@ -11,6 +11,7 @@
 import tempfile
 import unittest
 
+from Bio import bgzf
 from Bio import File
 from Bio._py3k import StringIO
 
@@ -67,6 +68,23 @@ def test_undohandle_read_block(self):
             h.close()
 
 
+class RandomAccess(unittest.TestCase):
+
+    def test_plain(self):
+        with File._open_for_random_access("Quality/example.fastq") as handle:
+            self.assertTrue("r" in handle.mode)
+            self.assertTrue("b" in handle.mode)
+
+    def test_bgzf(self):
+        with File._open_for_random_access("Quality/example.fastq.bgz") as handle:
+            self.assertIsInstance(handle, bgzf.BgzfReader)
+
+    def test_gzip(self):
+        self.assertRaises(ValueError,
+                          File._open_for_random_access,
+                          "Quality/example.fastq.gz")
+
+
 class AsHandleTestCase(unittest.TestCase):
 
     def setUp(self):